Skip to content

Building Giraph 1.1.0 For CDH 4.7.0

Jump to bottom
Dwayne Pryce edited this page Jun 17, 2014 · 9 revisions

Instructions for Building Giraph 1.1.0 for CDH 4.7.0

Warning: Currently, the Giraph 1.1.0-SNAPSHOT is broken for EdgeInputFormats - make sure you get the right commit hash before you build to avoid any bad behavior.

For information on building giraph see https://giraph.apache.org/build.html

Building giraph for your cluster can be difficult if your configuration is not specifically supported by giraph, to help we've provided steps we had to take to build giraph on two of our cluster configurations.

Getting Giraph with Git

$ git clone 
$ git reset --hard f0b6cddd3e255e9e31e3afe22d0f69b3c046d554

Building for YARN on Hadoop 2.0.0-cdh4.7.0

Download the giraph source, and from the giraph directory run:

mvn -DskipTests -Dhadoop.version=2.0.0-cdh4.7.0 -Phadoop_yarn clean install

Building for Map Reduce v. 1 (MRv1) on Hadoop 2.0.0-cdh4.7.0

Download the giraph source and add the following to the profiles section of the main pom

    <profile>
        <id>cdh4.7.0</id>
        <properties>
            <hadoop.version>2.0.0-cdh4.7.0</hadoop.version>
            <munge.symbols>HADOOP_1_SECRET_MANAGER</munge.symbols>
            <!-- TODO: add these checks eventually -->
            <project.enforcer.skip>true</project.enforcer.skip>
            <giraph.maven.dependency.plugin.skip>true</giraph.maven.dependency.plugin.skip>
            <giraph.maven.duplicate.finder.skip>true</giraph.maven.duplicate.finder.skip>
        </properties>
        <dependencies>
            <dependency>
                <groupId>commons-net</groupId>
                <artifactId>commons-net</artifactId>
            </dependency>
            <dependency>
                <groupId>org.apache.hadoop</groupId>
                <artifactId>hadoop-mapreduce-client-shuffle</artifactId>
                <version>${hadoop.version}</version>
            </dependency>
            <dependency>
                <groupId>org.apache.hadoop</groupId>
                <artifactId>hadoop-mapreduce-client-jobclient</artifactId>
                <version>${hadoop.version}</version>
            </dependency>
            <dependency>
                <groupId>org.apache.hadoop</groupId>
                <artifactId>hadoop-mapreduce-client-hs</artifactId>
                <version>${hadoop.version}</version>
            </dependency>
            <dependency>
                <groupId>org.apache.hadoop</groupId>
                <artifactId>hadoop-mapreduce-client-core</artifactId>
                <version>${hadoop.version}</version>
            </dependency>
            <dependency>
                <groupId>org.apache.hadoop</groupId>
                <artifactId>hadoop-mapreduce-client-common</artifactId>
                <version>${hadoop.version}</version>
            </dependency>
            <dependency>
                <groupId>org.apache.hadoop</groupId>
                <artifactId>hadoop-common</artifactId>
                <version>${hadoop.version}</version>
            </dependency>
            <dependency>
                <groupId>org.apache.hadoop</groupId>
                <artifactId>hadoop-auth</artifactId>
                <version>${hadoop.version}</version>
            </dependency>
            <dependency>
                <groupId>org.apache.hadoop</groupId>
                <artifactId>hadoop-client</artifactId>
                <version>${hadoop.version}</version>
            </dependency>
        </dependencies>
    </profile>

Then you can run the following command.

mvn -DskipTests -Pcdh4.7.0 clean install
Clone this wiki locally