Release 0.4 Third Week and Course Wrap Up

I'm pretty sure this is the third week of Release 0.4. I finally got to tackle an issue regarding Redis! Truthfully, I wished I did more Redis related work as it seems interesting. My issue can be found here.

I didn't do much work, most of the heavy lifting was done by people who worked on the storage.js portion of the code and the people who worked on creating the post object and mapping the appropriate fields. The week before this, we had an issue with the queue and were not able to figure out what to write in order to output things when a worker was done processing a job.

I'm imagining after lots of frustration on the professor's part, he managed to figure out what the appropriate code was:

feedQueue.on('completed', callback)

From there it was all about writing a callback that dealt with the job and results

  feedQueue.on('completed', async (job, results) => {

    try {

      await Promise.all(results.map(result => storage.addPost(result)));

    } catch (err) {

      logger.error({ err }, 'Error inserting posts into database');

    }

A simple explanation of the code:

On completion of the worker processing the job from the queue, it will store each individual post object into the Redis server with a unique ID. Once all the posts from the job are added, a promise will be returned, if there is an error during this this, an error will be displayed.

When it came to using functional programming, I never really got to practice them. I learned about .map, .reduce and .filter in java, but we only used it once for a lab and most of the time I have still been using for loops. It was nice to be reminded of how to use them again.

Lastly, when working on my first PR for this project which was the feedParser, I've read about Promise.all, Promise.resolve, Promise.reject and the other variations while learning about async/await. I didn't really understand it, but I'm now grateful this came up on my issue as the professor was able to explain how to use it. 

For anyone, who's wondering about Promise.all(), it is used handle a bunch of Promises. In this scenario on the line starting with await, each posts being stored to Redis is an asynchronous task and may take longer or shorter than others, we're going to wait until all the posts are stored from the job before returning a promise.

Very last note, this blog marks the end of the OSD600 course. I have to say, this course exceeded any expectations I had. It was very highly recommended by the professor from my Data Structures and Algorithm course, Catherine Leung as she mentioned there was no other course like it and the experience you get from the course is whatever you put in. I can only repeat the words she said word verbatim. For others, if you happen upon this blog and are wondering whether you should take OSD600, do it! I've previously read reviews that students should already be involved in open source or they'll have a bad time, I came in with 0 knowledge and it was extremely fun and a great learning experience. I'd also like to thank the professor for teaching this course, his patience and for reviewing close to 350-400 pull requests from his class of 60 students. That must have been hell.