Expose Stats within Spark

[nerdynick]

What kind an issue is this?

• Bug report. If you’ve found a bug, please provide a code snippet or test to reproduce it below.
  The easier it is to track down the bug, the faster it is solved.
• Feature Request. Start by telling us what problem you’re trying to solve.
  Often a solution already exists! Don’t send pull requests to implement new features without
  first getting our support. Sometimes we leave features out on purpose to keep the project small.

Feature description

It'd be nice to be able to get ahold of the Stats object post write within the Spark pipeline. This would allow diagnosis of pipeline performance issues from not just the ES Server side but also on the Client side.

I think the easiest and simplest method would be to provide the Stats as a new Partition/RDD after the writes have all been preformed. This would allow the user to control how the stats are published respective of their platform. This would involve changes with the EsRDDWriter to record the stats post write. As well as the EsSpark and JavaEsSpark classes to return the data as apposed to returning nothing.

[jbaiera]

I agree that exposing the stats in Spark is a great idea, though I'm not sure how I feel about them being a resulting rdd afterward. Much of our code is laid out as though the ES connector is an action in Spark, thus running the job. That said, there is some demand for returning and RDD/Dataset of failed writes to Elasticsearch after trying to write to ES. We could probably support both with a transformation-like RDD/Dataset operation, but I think we should make sure there are no other more appropriate routes to returning the job metrics to users before going down that path.

[nerdynick]

If you wanted to go the route of returning the failed records. Then maybe we handle it similier to how the Kafka consumers do with the offsets. By extending the RDD with methods to grab the stats from it and making the RDD result set the failed records.

The other idea might be just to return a object of some kind, representing all the information, that isn't an RDD. And let the user do with it what they please. Weather that be a Spark transformation to continue the pipeline on or saving information off into a DB of some kind.

[glasperfan]

If there hasn't been any progress here, I'd be interested in tackling this. The latter idea from @nerdynick seems flexible (not hard to construct an RDD from it or provide a util for it) and it shouldn't need to alter the spark-sql write api. Stats land in ReportingUtils and increment the counters, but could also accumulate into an exposed, top-level stats.
Agreed that failed records seem suited to a transformation output, but that could be handled separately from exposing stats

[jbaiera]

I can confirm that there hasn't been any work done on this yet from at least my end. Having looked at the issue again, I'm still not sure what the best route to follow would be from an API perspective. Perhaps if we had some information on how other integrations in the Spark ecosystem manage statistics, it would put my paranoia about modifying the main api to rest.