MapReduce is a programming model and an associated implementation for processing and generating large data sets. Users specify a map function that processes a key/value pair to generate a set of intermediate key/value pairs, and a reduce function that merges all intermediate values associated with the same intermediate key . In my case the map and reduce functions are passed to the workers through the plugins (.so files) . The worker and master talk via rpc present in the protocol package.
- Anyone who found this interesting , I will highly reccommend reading the official google research paper on MapReduce here .
- MIT also has a good lab reduce lab here
Running my demo with the simple wc.go Everything cann be run using Make
- generate the plugin.so file if its not generated The plugin can be changed by writing a new plugin and changing the pluginfile in Makefile
make plugin
- Run the master
make master
- Run the worker in the other terminal instance
make worker
The input to the master is given by .*txt which is essentially providing all the .txt files in the current directory