-
Notifications
You must be signed in to change notification settings - Fork 48
Use Spark to count the number of transactions from files containing Bitcoin Blockchain data
This is a Spark application demonstrating some of the capabilities of the hadoopcryptoledger library. It takes as input a set of files on HDFS containing Bitcoin Blockchain data. As output it returns the total number of inputs of transactions found in the blockchain data. It has successfully been tested with the Cloudera Quickstart VM 5.5 and HDP Sandbox 2.5, but other Hadoop distributions should work equally well. Spark 1.5 was used for testing.
See here how to fetch Bitcoin blockchain data.
After it has been copied you are ready to use the example.
Execute
git clone https://github.com/ZuInnoTe/hadoopcryptoledger.git hadoopcryptoledger
You can build the application by changing to the directory hadoopcryptoledger/examples/spark-bitcoinblock and using the following command:
../gradlew clean build integrationTest
This will execute also the integration tests validating the readiness of the application
Make sure that the output directory is clean:
hadoop fs -rm -R /user/bitcoin/output
Execute the following command (to execute it using a local master)
spark-submit --class org.zuinnote.spark.bitcoin.example.SparkBitcoinBlockCounter --master local[8] ./build/libs/example-hcl-spark-bitcoinblock-0.1.0.jar /user/bitcoin/input /user/bitcoin/output
After the Spark job has completed, you find the result in /user/bitcoin/output. You can display it using the following command:
hadoop fs -cat /user/bitcoin/output/part-00000
Blog: https://snippetessay.wordpress.com/2016/04/17/using-apache-spark-to-analyze-the-bitcoin-blockchain/
Understanding the structure of Bitcoin data:
Blocks: https://en.bitcoin.it/wiki/Block
Transactions: https://en.bitcoin.it/wiki/Transactions