Skip to content
This repository was archived by the owner on Sep 29, 2025. It is now read-only.

Analyse Litecoin data using Apache Spark

Jump to bottom
Jörn Franke edited this page Nov 12, 2017 · 3 revisions

This is a Spark application demonstrating some of the capabilities of the hadoopcryptoledger library. It takes as input a set of files on HDFS containing Litecoin Blockchain data. As output it returns the total number of transactions found in the blockchain data. It has successfully been tested with the Cloudera Quickstart VM 5.5 and HDP Sandbox 2.5, but other Hadoop distributions should work equally well. Spark 1.5 was used for testing.

Litecoin data differs from Bitcoin data in the sense that it uses a different hash function for the Proof-of-Work (Scrypt instead of SHA-256), which is irrelevant for analysis. Furthermore, it introduced certain features, such as segregated witness, earlier. As such, this example differs only in the configured network magic for Litecoin instead of Bitcoin.

Getting blockchain data

See here how to fetch Litecoin blockchain data.

After it has been copied you are ready to use the example.

Building the example

Execute

git clone https://github.com/ZuInnoTe/hadoopcryptoledger.git hadoopcryptoledger

You can build the application by changing to the directory hadoopcryptoledger/examples/spark-scala-litecoinblock and using the following command:

sbt clean assembly test it:test

This will also execute the integration tests

You will find the jar "example-hcl-spark-scala-litecoinblock.jar" in ./target/scala-2.10

Running the example

Make sure that the output directory is clean:

hadoop fs -rm -R /user/litecoin/output

Execute the following command (to execute it using a local master)

spark-submit --class org.zuinnote.spark.litecoin.example.SparkScalaLitecoinBlockCounter --master local[8] ./target/scala-2.10/example-hcl-spark-scala-litecoinblock.jar /user/litecoin/input /user/litecoin/output

After the Spark job has completed, you find the result in /user/litecoin/output. You can display it using the following command:

hadoop fs -cat /user/litecoin/output/part-00000

More Information

Understanding the structure of Bitcoin data (Litecoin is very similar):

Blocks: https://en.bitcoin.it/wiki/Block

Transactions: https://en.bitcoin.it/wiki/Transactions

Generic information about Litecoin: https://en.wikipedia.org/wiki/Litecoin

Clone this wiki locally