OSD700 Release 0.7

I tried to become a fullstack developer for this release. It... went better than I thought somehow? For this release I finally got to work on some front end. I remembered again why I like the back end so much more.

Thankfully, this release spanned 3 weeks instead of the normal 2 we're normally allotted. The first week I spent relearning how to use React and also learning how to use Material-UI. We're using Material instead of Bootstrap because some students in the class have developed a severe allergic reaction to Bootstrap. 

Author Result Component

Issue can be found here

I'm ashamed to admit, it took me at least 3+ hours just to make the search result component which can be found here (the time didn't include the design by @agarcia-caicedo). I'm very grateful I didn't have to design the component as well or we might not even have a component at all, also big thank you to @cindyledev for her reviews and suggestions as it made the process a lot quicker. 

Attaching the MyFeeds Component to the Backend

Issue can be found here

Wow, this took a whole day. Prior to this release, I saw our professor @humphd submit a PR to refactor some previously merged React code from class components to functional components and didn't understand what was going on... since it was front-end stuff. 

Well... a day was spent learning what functional components are(I learned React the class component way) and all the changes the React community have implemented. I read about how they are trying to steer people towards using React hooks instead of the previously used class components(although they are still supported). It took a while to understand (their docs are extremely helpful), still a newbie, but I understand it a lot more and have to admit, it is much better than their previous class components. Thank you to @Grommers00 for his work on the backend code changes and @Silvyre for creating the component + quick review.

Automatic Deployment to Staging on Master

Issue can be found here

I spent an afternoon with a frequent collaborator @manekenpix to figure out how this all works. It was pretty interesting as we thought it would take a very long time to get this working, but after 2 hours we had a simple server that was listening to issues filed in a private Github repository and would output a message. The next step would be to automate all the shell commands that are currently being manually executed by him whenever we want Staging to be updated. I am just hoping he gets to work on more issues aside from deployment because he has expressed how much he enjoyed coding.

Now, automatic deployment can range from being really simple to extremely complex. We can have an app that experiences some downtime as the app is updating to being really complicated where we apply the Green/Blue model and minimize any downtime.

Green/Blue - Two identical versions(Green and Blue) of the app will be available, one sits idle as the other one is running. When we bring down the one that is running (Green), we direct traffic to the other(Blue). This minimizes whatever downtime is being experienced as the current files in Green are being deleted, updated to the latest updates of the app from the GitHub repo and Green's containers are being spun up. Once Green is up and running again, we direct traffic back to the newly updated Green and idle Blue again. In this case the Blue can also act as a back-up which we can revert to if there are changes breaking the app or if Green is somehow experiencing issues.

Switch our app from REST to GraphQL

Issue can be found here

We implemented GraphQL and it works, but only if you go to http://localhost:3000/playground or https://dev.telescope.cdot.systems/playground. Out of all the issues this one was the most frustrating. I thought we were using Gatsby because it works well with GraphQL, I didn't know how it was going to work well, but it would just kind of work. This wasn't the case, we had to install an extra plugin called gatsby-source-plugin, It took a whole day for me to understand the changes I was making in gatsby-config.js 😂.

Anyways, this issue was so we can use all the queries we built for GraphQL to be used in the front end code too. A PR is ready to go which specifically addresses this issue... however moving forward some experimenting to understand how to use the data properly as well as what Static Query/ Page Query/ Static Query Hook are is going to be needed in order to use Gatsby properly with GraphQL

Overall, this release was a fun way to move away from just working on back end, refreshing/updating my knowledge on front end stuff and working on issues that were in between the two. I can't design things to save my life, but I can probably implement the design in React! Since I can do back end and front end things, I'm now a full stack developer right? Probably not.

For 0.8 alongside triaging issues, my goal is to try to get Kubernetes/minikube up for Production.

Fullstack Developer Wanted??

I don't get these job postings.

There was apparently an infamous post written on Medium by a big named guy declaring Fullstack is dead. 

After working on Telescope, I think so too. The field is so broad now with ever increasing amounts of technology a developer is supposed to know. A typical fullstack developer posting on sites goes as follows

We're looking for a fullstack developer with the following experience:

• Node.js + Express/Java/Golang/Python/.Net
• Javascript, CSS, HTML5
• Angular/React + Redux/Vue
• Docker
• No SQL: MongoDB/ Cassandra/ DynamoDB
• SQL: Oracle/ MySql
• AWS (may or may not include serverless functions)
• GIT

Bonus if you have experience with the following:

• Redis/Memcached
• User Experience design
• GraphQL
• CI/CD: Travis, Circle
• Kubernetes
• JWT/Auth0

WTF? One of the students in the class has spent pretty much a semester if not two, just learning and trying to implement SSO for Telescope. I mean I understand this is a wishlist, but this is insane. You might as well hire my Open Source Prof, @humphd at this rate. Thanks.  

OSD700 Release 0.6

Worked on a few issues for Telescope for this release:

GraphQL documentation for Telescope
Issue can be found here

This was fun, I knew nothing about GraphQL going into this. By the end of it I was even hacking away at nested queries on my own branch which we haven't even implemented yet in Telescope. I always shied away from documentation, because I rather be coding. I guess it is true. To see if you've really learned something, you should be able to explain or teach it to someone else.

GraphQL filters for Telescope
Issue can be found here

Aside from documenting on how to use GraphQL, I also took on an issue which required me to rewrite some queries to allow filtering and support future search functionality for the front end. This taught me some pain points about GraphQL as I always assumed it could do stuff like a traditional database, fro example: select * from posts where posts > provided date or something along those lines. GraphQL cannot or rather is unable to support this without installing another library, I ended up writing my own logic to do filtering and pagination.

On a side note, I also learned people can publish scalars(GraphQL typing) in packages for other people to download and use.

Include logic to filter inactive feeds and invalidate inactive feeds for Telescope
Issue can be found here

Another issue I started over the Christmas weekend and finally finished. This went through a few iterations and in the end it was suggested to scrap the current code written in favor of a more Redis oriented solution.

Refactor promises for plumadriver
Issue can be found here

I was suggested this issue by our prof, since I did quite a bit of work on refactoring promises for Telescope. It was an interesting experience reading typescript code and contributing to another repository after a few months of just working on Telescope.

GraphQL Nested Queries

The whole point of GraphQL is its flexibility, I can view all the authors in the database and then I can add an additional query that can display all the books by the one author, we call these nested queries. I recently spent an afternoon + evening with @manekenpix to take a look at nested queries in GraphQL for the Telescope project.

We currently have a schema like below

  # 'Feed' matches our Feed type used with redis
  type Feed {
    id: String
    author: String
    url: String
    posts: [Post]
  }  # 'Post' matches our Post type used with redis
  type Post {
    id: String
    author: String
    title: String
    html: String
    text: String
    published: String
    updated: String
    url: String
    site: String
    guid: String
  }

Notice feed can also return an array of Post,  to allow nested queries, we have to define them in resolvers after the Query:

module.exports.resolvers = {
  Query: {
    //Queries are here
  },
  Feed: {
    posts: async parent => {
      const maxPosts = await getPostsCount();
      const ids = await getPosts(0, maxPosts);
      const posts = await Promise.all(ids.map(postId => getPost(postId)));
      const filteredPosts = posts.filter(post => post.author === parent.author);
      return filteredPosts;
    },
  },
};

What the above code does is get all Posts in the database, then filter the Posts only returning Posts that have the same author as the returned value of the feed author. For example if I'm running the following query in GraphQL

{
  getFeedById(id: "123") {
    author
    id
    posts {
      title
    }
  }
}

and the author name is Marie, the parent parameter that is provided to the nested query (posts) will be the results of the getFeedById which in this case the author name is Marie.

Real life data using a classmate of mine:

OSD700 Release 0.5

As part of 0.5 I was working working mainly on two issues and got a chance to help someone start contributing to Telescope.

Async/Await

I've blogged a bit about using async/await to replace our Promise code in Telescope. I started the working during the winter break and was finally able to get that merged this week. The issue actually took a while as it spanned across ~15 files in Telescope and had me refactor functions and tests at the same time, which admittedly was pretty scary. I can say I know how to use async/await a bit better, but there's still a long road ahead!

Kubernetes(minikube)

My other issue I've been working on is a collaboration between me and another classmate @manekenpix to deploy Kubernetes(minikube) on a site for Telescope at http:://dev.telescope.cdot.systems/planet. We've had success being able to deploy services and even got the ingress to work on our own machines locally. However after 5 hours of sitting down and lots of expletives yelled at the computer, we had an issue when trying to deploy it on the machine CDOT has prepared to host Telescope. We forgot minikube runs using a vm on the computer, exposing the service and deployment only really exposes it to the computer the vm is running on. After a bit of researching and asking around on the slack channel we have decided to try a Bridged connection to expose the vm to outside traffic. We're crossing our fingers to have this for 0.6 (hopefully).

Helping a new contributor

Lastly, our professor Dave Humphrey has been actively recruiting students from his other classes to participate on Telescope (where was this teacher when I started learning web development). Which I think is an amazing idea as they gain experience in filing/fixing issues, receiving feedback and just collaborating with other programmers on an open source project. One student took on a great starter issue to standardize the error code on the project. I was kind of a mentor in helping the contributor get their code merged. This gave me flashbacks to OSD600 where our professor pretty much spent the whole semester teaching git and helping students with their git problems. Long story short, the student was able to get their PR merged and is happily taking on another issue. Git is hard and it is even more so when things land daily if not every few hours, the student admitted he used git before, but wasn't used to the pace at which Telescope moved.

The mentoring also taught me something, our professor has started to emphasize the importance of submitting a PR with some work completed instead of a full fledged PR. This way if their current work is starting to go sideways the community can direct the contributor to the correct path, preventing them from going further down the wrong path. For example, the contributor I was helping out kept trying to rebase, apply their changes then commit to their PR all in one go and this kept failing. Instead, I asked the contributor to:

1. rebase their PR and drop any of the unrelated commits, push code to their PR. At this point we'd review and see what other changes we need to make, such as do we have to use any files from master to the working branch because a file on the working branch is too far gone?
2. if the current status of the PR looked good, we'd apply their changes to fix the issue and review to see what other changes we need to make

This approach worked a lot better and the contributor got their PR merged today!

In hindsight, I think I've become a better programmer. 4/5 months ago I was attempting to enhance another person's simple note taking app on github.

Async Await and Promises

As a continuation of my PR for Telescope, I thought I should talk a bit about async/await and the old way of using return new Promise(). Here are a few examples of do's and don'ts:

// Async functions return promises, no need to add await 
// DON"T DO
async function returnsPromise(){
  return await promiseFunction();
}

// DO
async function returnsPromiseFixed(){
  return promiseFunction();
}

//---------------------------------------------------------------------------

// Don't use await when function is not async 
// DON"T DO
function noAsync(){
  let promise = await promiseFunction();
}

// DO
async function noAsyncFixed(){
  let promise = await promiseFunction();
}
//---------------------------------------------------------------------------

// Writing errors
async function f() {
  await Promise.reject(New Error("Error"));
}

// SAME AS
async function f() {
  throw new Error("Error");
}
//---------------------------------------------------------------------------
// Use try catch to wrap only code that can throw
// DON"T DO
async function tryCatch() {
  try { 
    const fetchResult = await fetch();
    const data = await fetchResult.json();
    const t = blah();
  } catch (error) {
    logger.log(error);
    throw new Error(error);
  }
}

// DO
async function tryCatchFixed() {
  try { 
    const fetchResult = await fetch();
    const data = await fetchResult.json();
  } catch (error) {
    logger.log(error);
    throw new Error(error);
  }
}
const t = blah();

//---------------------------------------------------------------------------

// Use async/await. Don't use Promises
// DON"T DO
async function usePromise() {
  new Promise(function(res, rej) {
    if (isValidString) {
      res(analysis);
    } else {
      res(textInfo);
    }
    if (isValidStrinng === undefined) {
      rej(textInfo);
    }
  })
}

// DO
async function usePromiseFixed() {
  const asyResult = await asyFunc()
}
// --------------------------------------------------------------------------

// Don't use async when it is not needed... Don't be overzealous with async/await
// For example the sentiment module we're using is not an async function
// DON"T DO
module.exports.run = async function(text) {
  const sentiment = new Sentiment();
  return Promise.resolve(sentiment.analyze(text));
};

// DO
module.exports.run = function(text) {
  const sentiment = new Sentiment();
  return sentiment.analyze(text);
};

// --------------------------------------------------------------------------

// Avoid making things too sequential
// DON"T DO
async function logInOrder(urls) {
  for (const url of urls) {
    const response = await fetch(url);
    console.log(await response.text());
  }
}

// DO
async function logInOrder(urls) {
  // fetch all the URLs in parallel
  const textPromises = urls.map(async url => {
    const response = await fetch(url);
    return response.text();
  });

  // log them in sequence
  for (const textPromise of textPromises) {
    console.log(await textPromise);
  }
}

// --------------------------------------------------------------------------

// Examples
// refactor following function:

function loadJson(url) {
  return fetch(url)
    .then(response => {
      if (response.status == 200) {
        return response.json();
      } else {
        throw new Error(response.status);
      }
    })
}

// Solution:
function loadJson(url) {
  let fetchResult = await fetch(url);
  if (fetchResult.status == 200){
    let json = await fetchResult.json();
    return json;
  }

  throw new Error(fetchResult.status);
}

// refactor to use try/catch
function demoGithubUser() {
  let name = prompt("Enter a name?", "iliakan");

  return loadJson(`https://api.github.com/users/${name}`)
    .then(user => {
      alert(`Full name: ${user.name}.`);
      return user;
    })
    .catch(err => {
      if (err instanceof HttpError && err.response.status == 404) {
        alert("No such user, please reenter.");
        return demoGithubUser();
      } else {
        throw err;
      }
    });
}

demoGithubUser();

// Solution:
async function demoGithubUser() {
  let user;
  while(true){
    let name = prompt("Enter a name?", "iliakan");
    try {
      user = await loadJson(`https://api.github.com/users/${name}`)
      break;
    } catch (err) {
      if (err) {
        alert("No such user, please reenter.");
        return demoGithubUser();
      } else {
        throw err;
      }
    }
  }
}

// Call async from non-async
async function wait() {
  await new Promise(resolve => setTimeout(resolve, 1000));

  return 10;
}

function f() {
  // ...what to write here?
  // we need to call async wait() and wait to get 10
  // remember, we can't use "await"
}

// Solution:
function f() {
  wait().then(result => alert(result));
}

Telescope Issue-525

The story of my PR for issue-525.

I can't believe I'm making gifs at 4 in the morning.

Kubernetes Pt3

*Blessed*

Thank you @manekenpix, I still have no idea how to fix all the problems we came across, but let us just enjoy this for now.

Kubernetes Pt2

In the previous post we used kubectl command lines to deploy. However we can create .yaml configuration files and have kubectl create them that way also

the .yaml file will have the following

apiVersion: (name)

kind: Deployment

metadata:

  name: (appName)

  labels:

    app: (imageTag)

spec:

  replicas: (replicaNumber)

  selector:

    matchLabels:

      app: (imageTag)

  template:

    metadata:

      labels:

        app: (imageTag)

    spec:

      containers:

      - name: (imageTag)

        image: (dockerImage)

        ports:

        - containerPort: (portNumber)

Then enter the following command:

kubectl create -f (.yaml file)

I pulled an example from the edx Kubernetes course using the nginx image to deploy a webserver

apiVersion: apps/v1

kind: Deployment

metadata:

  name: webserver

  labels:

    app: nginx

spec:

  replicas: 3

  selector:

    matchLabels:

      app: nginx

  template:

    metadata:

      labels:

        app: nginx

    spec:

      containers:

      - name: nginx

        image: nginx:alpine

        ports:

        - containerPort: 80

This will deploy an app named webserver replicated across three pods.

We can also define a appName-svc-.yaml file to expose our service with the following content:

apiVersion: (get this value from running kubectl api-version)
kind: Service
metadata:
  name: web-service
  labels:
    run: web-service
spec:
  type: (serviceType)
  externalName: (externalLink) *Use this field if serviceType is set to ExternalName
  ports:
  -  port: (portNumber)
     protocol: TCP
  selector:
    app: (imageTag)

Then enter the following command:
kubectl create -f (appName)-svc.yaml

serviceType can be any of the below:

1. LoadBalancer - if the cloud provider Kubernetes is running on provides load balancing.
2. ClusterIP - can only reach the service only from within the cluster
3. NodePort - creates a ClusterIP and NodePort service will route to it. Allows access from outside the cluster by using NodeIP:NodePort
4. ExternalName - maps the service to the contents of the externalName field

Also pulled from the edx Kubernetes course:

apiVersion: v1
kind: Service
metadata:
  name: web-service
  labels:
    run: web-service
spec:
  type: NodePort
  ports:
  -  port: 80
     protocol: TCP
  selector:
    app: nginx

Kubernetes

Containers became all the rage nowadays, I have 0 experience with either Docker or Kubernetes. This post serves to kind of explain some concepts Kubernetes for myself.

Pods - can be made up of one or more containers. Pods can also be replicated horizontally to allow scaling of an app.

Deployments are used to manage pods, to deploy a pod we use the following line

kubectl create deployment (appName) --image=(imageName)

kubectl get deployments - will display all current deployments

kubectl get pods - will display all pods 

kubectl get events - will display all the things that have happened, such as new pods

Although we have created a deployment for our pod, it is only accessible within the Kubernetes cluster. A Service is enables access to the deployed App, to create a Service we have to use the following command

kubectl expose deployment (appName) --name=(serviceName) --type=LoadBalancer --port=(portNumber)

*if the --name=(serviceName) flag is not provided, the service will default to the appName
*--type= can be any of the below:
LoadBalancer - if the cloud provider Kubernetes is running on provides load balancing.
ClusterIP - can only reach the service only from within the cluster
NodePort - creates a ClusterIP and NodePort service will route to it. Allows access from outside the cluster by using NodeIP:NodePort
ExternalName - maps the service to the contents of the externalName field

We can verify the Service has been created by using the following command:

kubectl get services , this will display all the Pods exposed 

minikube service (serviceName) , will launch the service within the pod.

Technically the steps we need to follow to deploy an app on Kubernetes

1. Create a deployment to manage (kubectl create deployment (appName) --image=(imageName))

2. Expose the deployment kubectl (kubectl expose deployment (appName) --name=(serviceName) --type=LoadBalancer --port=(portNumber))

3. Run the service (kubectl service (serviceName))

To replicate the pods we use the following command

kubectl scale deploy (appName) --replicates=(replicateNumber)

On a side note, this also lets us manage some deployments on the fly. Say our current image is not compatible with other images, we can change the version by using the following command.

kubectl set image deployment (appName)=(imageName)

Kubernetes tracks histories of all changes made to the deployment, such as when changing the image for a deployment. They can be viewed with the following command

kubectl rollout history deploy (appName)

When changes are made to the image, Kubernetes will automatically scale down replica sets of the deployment with the old image and automatically spin up the same number of replicas for deployment with the newer one. We can verify this by using

kubectl get rs -l app=(appName)

To rollback changes made to a deployment we use the following command. The revisionNumber can be any of the ones listed when running the command kubectl rollout history deploy (appName)

kubectl rollout undo deployment (appName) --to-revision=(revisionNumber)

When rolling back changes, a new revision will be made and it will also remove the revision number of the one we rolled back to. For example I initially deployed with an image of version 1.15 and changed the image to version 1.16. There should be a total of 2 revisions: 

• 1 (my initial image of version 1.15)
• 2 (my current image of version 1.16)

If I roll back to revision 1 with the above command, a new revision will be added to the table 3, and revision 1 will be removed. My history will now look like the following:

• 2 (image of version 1.16)
• 3 (image of version 1.15, I rolled back to)

Kubernetes tracks up to 10 revisions for your rollback pleasure.

To delete the deployment use the following command
kubectl delete deployments (appName)

So I Added New Linting Rules

An issue was filed for Telescope to address consistency and proper use of async/await + promises and fix them. I added new linting rules. Ran an npm test:

Oh my god.