alantanlc / map-reduce

Map Reduce Framework in Python

Map Reduce

Components

User Program

Tasks:

1. Fork master and workers

Master

Variables:

1. List of workers
2. List of files

Functions:

1. constructor
2. initializes workers list
3. initializes files list
4. assignMap function
5. e.g. each MapWorker takes an input file
6. assignReduce function
7. e.g. each ReduceWorker takes an intermediate file
8. run function
9. for each worker, execute

AbstractWorker

Functions:

1. abstract execute function
2. abstract read function (read)
3. abstract emit function (write)

AbstractMapWorker

Variables:

1. input_filename
2. intermediate_filename

Functions:

1. implements execute function which calls abstract map function
2. abstract map function

WordCountMapWorker

Functions:

1. implements map function
2. read file
3. split content into words
4. generate word-count map
5. implements emit function
6. writes results to intermediate

AbstractReduceWorker

Variables:

1. output_filename

Functions:

1. implements execute function which calls reduce function
2. abstract reduce function

WordCountReduceWorker

Functions: 1 .implements reduce function

1. read file
2. collect results
3. implements emit function
4. writes results to file