I can’t quite find the blog post but I saw someone do a blog post using AWS’ map reduce on multiple servers to process a dataset… and then they redid their pipeline using bash, awk, and maybe grep and a single 8-core machine did it 100 times or so faster.
I can’t quite find the blog post but I saw someone do a blog post using AWS’ map reduce on multiple servers to process a dataset… and then they redid their pipeline using bash, awk, and maybe grep and a single 8-core machine did it 100 times or so faster.
Edit: found it https://adamdrake.com/command-line-tools-can-be-235x-faster-than-your-hadoop-cluster.html