The Creative Network Session 8— Server less architecture, Lambda and Gateways

Previous sessions

1. Introduction session
2. Setting up redux
3. Home Page with React
4. Setting up GraphQL
5. GraphQL schema V1
6. Integrating GraphQL in React with Lokka
7. Building photo upload with AWS Lambda and S3

In the previous session we were trying to experiment some ideas of how to architecture the core photo upload feature of our site. We thought we had some good ideas and ready to roll. But apparently after Avi woke up in the morning he thought of something even BETTER. In this session we went on to implement the improved version.

As a reminder here is the previous version of our architecture. This feature is for photo upload:

Avi pointed out that there were several things he didn’t like about this architecture. The most outstanding issue is the resize process. In this setup, every image being uploaded are automatically resized to a pre-determined set of images and stored. Resizing images are resource intensive and most of the time the resized version won’t be consumed by clients until later or from another API endpoint. So what’s the point of clogging the system with resize at the moment of upload?

So, we have an improved V2.

The solution is on-demand photo resizing. In the architecture above, client uploads an image we we store this image in S3. At the same time we trigger a lambda function that calls our API cluster and persist image url in our RDS server, at this point there is no resize AT ALL. The resize happens when the client actually REQUEST for a certain size of the images, this request would come in at an AWS API Gateway, then we triggers another lambda function. At this point we only have the original size. The lambda would check if we have that particular size available, and do one of two things:

1) If the requested image size IS NOT available, do the resize NOW, store it in S3 and return the response to client.

2) If the requested image size IS available, simply return the image location and DO NOT do resize.

We would then cache the image url with a 301 redirect rule so that once the client have requested the image once it would never actually trigger this lambda function again.

The biggest advantage of this architecture:

• The initial upload process will be MUCH faster and much less resource intensive, since we don’t do resize here and simply store in S3.
• The resize happens on demand, client can get ANY size they want without change in server code. Also resources are only consumed when a particular size of image is being explicitly requested by client.
• The lambda function/resize only gets triggered ONCE. The resized image url will then be cached with 301 redirect and which makes sure any subsequent request for same image with same size will have no performance hit at all.

We started with this implementation last night and will continue in the next few sessions. Here is the Code.

One complication with our current approach is that we use Node Sharp to resize image. Sharp needs to compile a native extension to work and in order to deploy sharp with lambda function, we’ll need to compile in a Linux env instead of our Mac env. We’re planning to solve this by using Docker and Terraform in the next few sessions.

Stay tuned and if you’re interested in the live streaming you can subscribe here!