README

This guide describes how to use the source code developed for the study in:

Efficient Parallel Set-Similarity Joins Using MapReduce.
Rares Vernica, Michael J. Carey, Chen Li
SIGMOD 2010 

The only requirement for running the code is a Hadoop cluster. It does not have to be a full-fledged cluster, a single-node pseudo-distributed installation of Hadoop is enough. For more details about starting a Hadoop cluster please see http://hadoop.apache.org/common/docs/current/quickstart.html The code has been compiled with Hadoop-0.17.2.1.

The XML files provided with the -conf argument above contain various configuration parameters. Using the configuration parameters, a user can specify the location of the data, the similarity function and threshold, the join attributes and other settings. Moreover the user can specify additional parameters in the command line using the -D option.

The default parameters and more details about each parameter are in:

hadoop/fuzzyjoin/resources/conf/fuzzyjoin/default.xml

All these parameters and other constants are defined in:

fuzzyjoin/src/edu/uci/ics/fuzzyjoin/Config.java
hadoop/fuzzyjoin/src/edu/uci/ics/hadoop/fuzzyjoin/FuzzyJoinDriver.java

The following directory structure is used:

|
|- raw
|- recordsbulk.00
|- recordsbulk.01
|- ...
|- records.1
|- records.2
|- ...
|- tokens.1
|- tokens.1.phase1
|- ...
|- ridpairs.1
|- ...
|- recordpairs.1
|- recordpairs.1.phase1
|- ...

The raw directory contains the original files, one record per line. The recordsbulk directory contains the original data where each record starts with an integer RID. The number after the directory name represents the copy number (00 is the original data, 01 is the first copy, etc.). The records directory contains the same data as the recordsbulk except that multiple copies are aggregated and data is balanced across nodes. The number after the directory name represents how many copies are aggregated (1 is of only one copy: recordsbulk.00, 2 is for two copies: recordsbulk.00 and recordsbulk.01, etc.). So records.n represents an increased dataset, where n denotes how many times the dataset was increased. For the rest of the directories the number after the directory name has the same meaning. The tokens directory contains the list of tokens. The ridpairs directory contains the RID pairs that fuzzy-join. The recordpairs directory contains the record pairs that fuzzy-join. The phase1 prefix that appears for some directories represent the output of the first MapReduce job for the tasks with two MapReduce jobs (i.e., tokensbasic and recordpairsbasic).

Bellow is a table with each task input and output directories:

By default the dataset is assumed to have one record per line. The fields of each record are delimited by ":". The first filed of each record is an integer RID. This settings can be changed in:

fuzzyjoin/src/edu/uci/ics/fuzzyjoin/Config.java

The dataset can be increased using the recordgenerate task:

hadoop/fuzzyjoin$ hadoop jar build/jar/fuzzyjoin-hadoop-1.0.0.jar \
  recordgenerate -conf resources/conf/fuzzyjoin/dblp.quickstart.xml \
  -Dfuzzyjoin.data.copy=10 \
  -Dfuzzyjoin.data.norecords=80

The fuzzyjoin.data.copy parameter specifies the number of times the dataset should be increased, while the fuzzyjoin.data.norecords parameter specifies the number of records in the original dataset (it is used to generate unique and increasing RIDs). All the following task also need to have the same value for the fuzzyjoin.data.copy parameter in order to use the increased dataset. This task is ran after running recordbuild and tokensbasic or tokensimproved on the original data. After this task, the recordbuild task needs to be ran (it cannot be skipped on the increased dataset):

hadoop/fuzzyjoin$ hadoop jar build/jar/fuzzyjoin-hadoop-1.0.0.jar \
  recordbalance -conf resources/conf/fuzzyjoin/dblp.quickstart.xml \
  -Dfuzzyjoin.data.copy=10
hadoop/fuzzyjoin$ hadoop jar build/jar/fuzzyjoin-hadoop-1.0.0.jar \
  fuzzyjoin -conf resources/conf/fuzzyjoin/dblp.quickstart.xml \
  -Dfuzzyjoin.data.copy=10

The source code is divided into two modules:

• fuzzyjoin: general fuzzy-join code
  • edu.uci.ics.fuzzyjoin: main memory fuzzy-join
  • edu.uci.ics.fuzzyjoin.similarity: similarity functions and filters
  • edu.uci.ics.fuzzyjoin.invertedlist: inverted lists index
  • edu.uci.ics.fuzzyjoin.recordgroup: alternatives for grouping records
  • edu.uci.ics.fuzzyjoin.tokenizer: tokenizes
  • edu.uci.ics.fuzzyjoin.tokenorder: alternatives for ordering tokens
• hadoop/fuzzyjoin: Hadoop specific fuzzy-join code
  • edu.uci.ics.hadoop.fuzzyjoin: main program
  • edu.uci.ics.hadoop.fuzzyjoin.datagen: classes for building records and increasing dataset size
  • edu.uci.ics.hadoop.fuzzyjoin.recordpairs: Stage 3
  • edu.uci.ics.hadoop.fuzzyjoin.ridpairs: Stage 2
  • edu.uci.ics.hadoop.fuzzyjoin.ridrecordpairs: alternative to Stage 2 and 3 where records are not projected
  • edu.uci.ics.hadoop.fuzzyjoin.tokens: Stage 1