PHP and Python unable to run tests.

[neclimdul]

Seems from #288 there is no app.json support which means buildpacks like php and python don't even work completely. There's no test provided by the buildpack and you're expected to use app.json and without support it doesn't work.

This is a blocker for these languages using systems like gitlab autodev which us herokuish for zeroconf testing.

[josegonzalez]

I have no idea how gitlab runs its "zeroconf testing", and to be honest, thats not really a herokuish problem. Dokku partially supports app.json manifests, but we do that on the Dokku-side, not within herokuish.

That said, if there is a way you can build this in that will replicate how it works on Heroku, I am all ears. Note that the app.json spec isn't even really maintained by them - in favor of a heroku.yml spec - so I don't know if this is the path forward to begin with.

[neclimdul]

sorry. I don't have a lot of experience with heroku so my limited ability to explain the problem may have got this off on the wrong foot.

First gitlab's test step is based around herokuish buildpack test is what lead me to herokuish.

https://devcenter.heroku.com/articles/heroku-ci#php is where I was looking to solve this problem. It seems to be the only method I've found for providing testing for php/python buildpacks.

So applying these things I see this:

root@93d393b92933:/tmp/app# cat app.json
{
  "environments": {
    "test": {
      "scripts": {
        "test": "phpunit"
      }
    }
  }
}
root@93d393b92933:/tmp/app# /bin/herokuish buildpack test
-----> PHP app detected
Selected buildpack does not support test feature

[josegonzalez]

Seems like if that test file does not exist, we could check for the existence of an app.json in the application, extract the test command, and run it in the right context.

If you'd like to contribute that, we'd almost certainly accept it (with the accompanying tests to herokuish so we don't accidentally drop the functionality of course).

[neclimdul]

Wouldn't it make sense to use the app.json script first and then fallback on the buildpack?

I'm willing to look at that but I have no idea where to start adding tooling to parse json and the like.

[josegonzalez]

I would do whatever heroku does in this case. It seems like if the buildpack has a test command, it uses that over the app.json, based on the docs you linked to. But if you are confused, I would check there.

As far as where to start, this location seems good, as it appears to be where the error you pasted comes from.

[neclimdul]

Comments like "For typical Ruby apps, you should not need to define scripts in app.json." would imply to me the opposite which makes sense. You should use explicit values over implicit defaults.

What about parsing json in bash? I would just use something like jq but I don't know how you treat dependencies like that.

[josegonzalez]

I don't know what heroku does in terms of order of operations, so you should test that with sample apps where that isn't clear.

As far as parsing in bash, you might be able to get away with injecting a small golang binary to grab the test command.

[neclimdul]

so the run process is pretty clear here https://devcenter.heroku.com/articles/heroku-ci-technical-detail-on-ci-lifecycle#step-5-the-test-run-is-executed

On each step first app.json is used. then buildpack is called.

so I've dug around and I'm just not clear on how to do any of that or how to develop or test herokuish. your irc channel has been pretty quite so not sure about fixing this myself.

[josegonzalez]

@neclimdul sorry, I don't open irc much. Would you be willing to join us on slack with any questions?

[neclimdul]

sure, request submitted.

[josegonzalez]

You'll need to check your email for the invite :)

[neclimdul]

I think your slackin is broken. slack.gliderlabs.com socket polling shows only 400 and the /invite POST returns a 403.

[josegonzalez]

I've jumped on irc (I am savant) in the #dokku room if you want to say hi there.

[VincentSurelle]

Hi,

How is this going ? I'm also trying Gitlab's Auto-Devops. And even if my app.json is complete, I can't test my PHP Application.

Any direction to get it work ?

Thanks ;)

[neclimdul]

@VincentSurelle I haven't had time to dig in to the solution, there is some learning I need to do to address it.

For the time being I'm using https://github.com/neclimdul/heroku-buildpack-php to get past that particular pipeline job. It provides a test script that calls composer test which works as long as you create a test script in your composer.json. Take a look at https://getcomposer.org/doc/articles/scripts.md if you're not familiar with how that works.

That build pack is not a long term solution though because I only plan on maintaining it until this issue is fixed.

[sekamaneka]

Any news on this? I am currently stack at this point.

[neclimdul]

@swilso793 what little time I had to look at this completely dried up. I did file an issue with gitlab since they are invested so maybe more eyes can get involved.

https://gitlab.com/gitlab-org/gitlab-ce/issues/47651

[josegonzalez]

If anyone from Gitlab is reading this, if you want this to work for your customers, you're going to have to implement it. Its a great idea, and I think it makes a lot of sense to include it, but I don't have time to work on this and I don't think the other maintainers do either.

If anyone is asking for "news", there won't be any real updates unless someone either:

• writes the pull request to implement the feature
• sponsors the work to implement the feature

Hope that makes sense.

[svengerlach]

We took @neclimdul's idea of a change in the php buildpack a little further and came up with a small merge request for heroku's official php buildpack (heroku/heroku-buildpack-php#289). I especially liked the idea of using "composer test" as an entrypoint for my tests. Let's see if the change gets merged.

[josegonzalez]

The PHP buildpack has support for bin/test as of heroku/heroku-buildpack-php#350.

If anyone would like to sponsor the work of adding app.json testing support, feel free to reach out.

[vincetse]

Just sent a PR (heroku/heroku-buildpack-python#982) to get this added to the Heroku Python buildpack. Hope it gets accepted.

[josegonzalez]

Going to close this as it's not really something we can fix in this project. We'll pull in new versions of each buildpack as they come in.