CS Journey's RSS Feed

Life's Great. Things Could Always Get a Whole Lot Worse - Stories from 2025.

Sun, 18 Jan 2026 00:00:00 GMT

Life’s great as it is. Every time I look back at the year that passed, I’m reminded that things could get a whole lot worse. 2025 hit a little different.

This isn’t to say that I didn’t come across happy stories in 2025. Maybe for another time.

A stark reminder that life’s been great despite a few challenges.

A friend whose saving had been depleted by her family on wasteful expenditure while her brother’s salary remained untouched.
A student who’s the sole breadwinner at his home after his dad. Working multiple jobs and taking on loans for an event only for the outcome to be less than ideal.
A random Google search to learn more about a man who provided me some information over a WhatsApp group. Turns out he’s a 83 year old gentleman helping out people all over India while being a caregiver for his daughter for over 30 years. He runs a WhatsApp group, patiently types out super long replies despite having tremors in his hands, get on calls to provide encouragement and words of support, advocates for rights and has played a role in changing laws. He was also a caregiver for his late wife during her battle with cancer - Amrit Sir 1 2

A friend (Doctor) who put her own treatment at hold because she’d been helping address financial constraints at home.
A conversation with a 74 year old mother who’s worried about the survival of her n year old son (PMI - Personal with Mental Illness) when she and her husband stop being around. They’re looking to move to a different city and exhaust their savings to live in an assisted living facility where their son will be taken care of after them.
A caregiver recounting the story of her son (unmarried) who believes he has to travel to a certain country to find his lost wife and kid.
Stumbled up this Reddit post.
A Doctor shared a story about his PMI who believed that she ruled a certain country and had to get there. She lives alone in India and manages by herself while her family lives abroad.
A mental rehabilitation nurse showing the ward with PMIs who had arrangements to live there for life (for various reasons).
Sharings from strangers in a WhatsApp group. Stories about a mom taking care of her son, stories of a mom and daughter admitted in psychiatric wards and so on.

A friend who’s been depressed, working a really stressful job, has monetary constraints but still had to be on a leave of abscence to tend to her own health. To top this multiple family members have been falling sick.
A fellow caregiver who’s been trying to get her brother treated from the last x years, but to no avail. She has since given up on the medical system and has been resorting to spirituality as a means to cope.
A Doctor shared the story of a patient who’s been confined to the walls of a hospital for the last 12 years but hasn’t been responding to Meds.
2 perfectly normal seeming friends who’ve experienced intense psychosis.
A friend’s sibling who needed to be admitted and undergo ECT (shock treatment) to treat MDD from a certain trauma.
A relative who’s been living in delusions at a time before new age medicines for the same. Violence at the mental rehabilitation facilities, shock treatment been through it all.
A relative’s young daughter who passed away in an unfortunate car accident.
A friend (Doctor) recalling patients suffering from cerebral palsy, still needing assistance in their 30s with basic hygiene, parents holding on to hope and having to break the news to them
A friend’s brother and his battle with addiction.
A student’s young relative and his recovery from hallucinations and being on the journey to become a Doctor. More power to him.
A student’s mom’s stint with talking walls.

Really puts things into perspective.

Song Analysis - Fix You by Coldplay

Mon, 12 Jan 2026 00:00:00 GMT

Lyrics copied verbatim from Genius.

General

Slow, Time Signature: 4/4
Major Scale Diatonic Chords
Short Chorus
Really picks up after the 2nd Chorus

Verse 1 and 2

When you try your best, but you don’t succeed
When you get what you want, but not what you need
When you feel so tired, but you can’t sleep
Stuck in reverse

And the tears come streamin’ down your face
When you lose somethin’ you can’t replace
When you love someone, but it goes to waste
Could it be worse?

Rhymes
Grouped similar beat and melodic triplets
- try your best, succeed, what you want, what you need
Repetition - when you, but you
Ambient chords
Verse 2 the harmonies are more prominent
Overall the pitch range is small. However there are a few big jumps in pitch
- when you -> try

Chorus 1

Lights will guide you home
And ignite your bones
And I will try to fix you

Similar beat & melodic structure
- guide you home
- ignite your bones
Addition of keys to the harmony
Song title used
Overall pitch range similar to the Verse

Song Analysis - Baarishein by Anuv Jain

Fri, 02 Jan 2026 00:00:00 GMT

Lyrics copied verbatim from Genius.

General

Slow, Time Signature: 4/4
Prominent vocals
Chord Progression: 4 chords I vi IV V from the Major Scale throughout the song
Feels like there’s a key change from Ab tere bina yahan meri saansein but after referring a few sites doesn’t seem to be the case
Chorus doesn’t repeat which is unusual for a pop song
Doesn’t loop back to the chorus, keeps progressing forward. Climax at the Bridge and fades in the outro.
Breathy exhales/donkey breathing at the end of few lines

Verse 1

Haule se, dheeme se
Mujhko baahon mein bhar lo na tum
Narm si saanson mai
Mujhko aahon main bhar lo na tum

Filled with rhymes (internal and strong)
Minimal harmonies - ambient sustained wobbly chords at the 1 beat
Simple beat (a shaker)

Chorus 1

Sun zara, mere paas aa
Ab baithe hain hum bhi yahaan
Dil ke darmiyaan
Baarishein hain, baarishein hain
Teri hi baaton pe maine saja li hai duniya yahaan
Dil ke darmiyaan
Baarishein hain, baarishein

Almost felt that it’s at the same intensity as the Verse
Chorus made stronger by repetition Dil ke darmiyaan Baarishein hain, baarishein hain
Repetition of the song’s name
Call to action? Sun zara, mere paas aa..
Scene setting Ab baithe hain hum bhi yahaan..
Exaggeration and metaphors
Some phrases have backing vocals. Eg: Sun zara, Baarishein hain

Bridge

Ab tere bina yahan meri saansein
Jaise bina nindiya ki raatein hain toh
Aur tu hi mere dil ki raza hai
Tere bina dil bhi khafa hai toh
Teri aankhon ka kajal na faile ab kabhi bhi
Tujhe itna pyaar doon, haan
Teri khushiyon ki khatir yeh duniya main meri
Ek pal mein waar doon main

Picks up a little after the Verse 2. 8 bars of just chords and rhythm before breaking into the Bridge
On referencing a few sites, here the first chord is a IV and he sings higher giving the illusion of a key change
Imagery and metaphor Teri aankhon ka kajal..

Outro

Ab bikhri teri ye zulfon se aankhein teri jab dikhti hain
Dikhta hai mujhe woh aasmaan
Ki kholun pankh main mere, ud jaaun main, kho jaaun main
Iss aasmaan main patangon ki tarha, haan

Simile - aankhein teri jab dikhti hain Dikhta hai mujhe woh aasmaan
Remarkedly lower energy from the bridge indicating the end

Song Analysis - Perfect by Ed Sheeran

Sat, 27 Dec 2025 00:00:00 GMT

Lyrics copied verbatim from Genius.

General

Slow, waltz like rhythm, 3/4
Triplets, prominent vocals
Time Signature: 6/8 because it fits better with the lyrics
Chord Progression: 4 chords I vi IV V from the Major Scale throughout the song
Ending the Chorus with Tonight

Verse

I found a love for me
Oh, darlin’, just dive right in and follow my lead
Well, I found a girl, beautiful and sweet
Oh, I never knew you were the someone waitin’ for me

Filled with rhymes
- me, lead, sweet, me (weak rhymes and strong rhymes)
- found - love, dive - right, love, love - follow, knew - you, never - were (weak rhymes)
Call to Action
- follow my lead
Harmonies are minimal. Can just hear a guitar

Pre-Chorus

‘Cause we were just kids when we fell in love
Not knowin’ what it was
I will not give you up this time
But, darlin’, just kiss me slow
Your heart is all I own
And in your eyes, you’re holdin’ mine

Stronger than the verse
Again, filled with rhymes (eyes - mine, slow - own etc)
kiss me slow, heart, in your eyes (imagery?)
Harmony addition: Dreamy/ambient textured Chords at the 1 beat

Chorus

Baby, I’m dancin’ in the dark with you between my arms
Barefoot on the grass while listenin’ to our favourite song
When you said you looked a mess, I whispered underneath my breath
But you heard it, “Darlin’, you look perfect tonight”

Vocals a lot higher. Verse and Pre-Chorus were kept low to create space for the Chorus.
Melodic tension
- Dancin in the dark - minor 2nd tension to root, 7th to root
- Barefoot on the grass - same as above
Vivid descriptions of the scene engaging multiple senses.
- between my arms, barefoot on the grass (touch)
- favourite song, whispered underneath my breath (sound)
Ending the Chorus with Tonight
Harmony - The ambient chords are a lot more prominent.

(WIP) Python cachetools - Source Code Walk

Mon, 08 Dec 2025 00:00:00 GMT

I’ve been wanting to get back to reading papers and open source code for some time now. As part of a workshop conducted by a friend, I’ve set a goal of grokking 12 pieces over the next 12 months.

A piece meets the criteria if
- It excites me (or)
- It’s around a problem I’m interested in (or)
- It has some side benefit (career or otherwise)

My goal for this week is to finalise at least 4 pieces from my bookmarks and notes.

Vatsal has created a small accountability group to ratchet up the pressure :D

And quoting his presentation, “If nothing works, you can just blackmail people to make them accountable.” :P

For the first piece, I’ve decided to start with a Python library that I find myself using often cachtools. I had a note as such: TTL Cache Python (timed dictionary) reminding me to check out the implementation.

It’s a nifty library that provides caching functionality out of the box. It’s similar to Caffeine or Guava in the Java world.

I generally start off by understanding the major data structures (thanks Antirez) and going through the Unit Tests. However, since this code base is really small, I’ve decided to get answers for the following questions that were on my mind instead.

How does it maintain global state for every permutation/combination of configuration?
How does it handle concurrency or does it just rely on the GIL?
TTL - Does it run a janitor in the background or is it on demand?
What sort of optimisation does it use to quickly delete expired items?
How are equivalent deep/pass by reference objects handled when creating the hashkey?
General re-usability patterns to handle different eviction policies.

Constraints are Real. For some Acceptance Comes after Hesitation.

Thu, 29 May 2025 00:00:00 GMT

I’ve been a cautious person in general for as long as I can remember. However, I’ve always felt that I’m immune to the realities of growing up. I felt I could circumvent these outcomes with time management, discipline and control. Incredibly naive of me.

Family getting older
- I can minimise the impact of time by controlling their lives towards healthier habits
Losing touch with friends/family once I move to a different city
- I will never tire. I could frequenly hop between Chennai and Bangalore
- I will create a rotation and talk to every friend group
Hadn’t seen death
Overly optimistic about the time taken to accomplish goals
- Will just work harder for a few more months and then I can go back to balanced life

Also, time and again I’ve failed to change myself, what made me think I could change others 😅?

This approach (call it Mr Sunshine or delusional living) hadn’t worked with teenage Abhirath and neither is it working now. Why do I cling to this mechanism? Have I just been trying to avoid the truth? I can’t have everything, heck even the few most important things in life are really difficult to acquire and maintain, it’s time to grow up.

Grandmother, Music and Memory

Sun, 25 May 2025 00:00:00 GMT

Music is powerful, music is lovely. So is my grandmother.

Music has this strange effect where it brings back associated memories of the time and context one heard it in. Why the heck does one remember melodies from years ago? :D

When I’m home with my family, it’s a given that I will be participating in a short drive for coffee. My grandmother decided to join us for a change. I fire-up some old Hindi hits on Spotify and enquire if she recognises them. Lo and behold, she murmurs out the name “Noor Jehan” and the song “Jawan Hain Mohabbat”.

She hadn’t heard of or heard the song in more than 20 years. Amogst other things and grandfather passing away - 2024 and 2025 haven’t been easy on her on mentally.

Yet, music prevailed.

Portfolio - Live Algo Trading Platform

Sat, 28 May 2022 00:00:00 GMT

This is a small demonstration of a project I’ve been working on. Mainly meant to showcase it to interviewers or clients :)

Index

Context
- Issues with Paid APIs
- Objectives
Features
Tech Stack

Context

Issues with Paid APIs

Most algotrading APIs are quite restrictive. You can only call them a few times per minute. We can fetch prices of multiple scripts at once at the cost of making our code complex. Your classes and functions would start needing an additional higher order iterator to deal with a single class/function acting on one security.
Real time prices -> via blocking web sockets (one thread or process will need to be dedicated for this). Historical prices via REST API. This undeniably increases code complexity.
Limits on the number of web socket connections -> Each algo of yours would run as a separate process. Brokers limit the number of web socket connections to 3 or 5. This stops us from running more processes.
Difficult to make it system agnostic (centralise using the cloud) given that in most cases we need our own custom handlers apart from the paid APIs.
Restrictive clients provided by service providers.
- Need to restart if a configuration change takes place.
- Difficult to parallelise.
Difficult to use multiple data providers.

Objectives

abstract away all the above mentioned complexities.
cater to a much larger scale by relying on in-house infrastructure.
smarter and faster client side code to better handle faults.
one unified system to handle data from different service providers.
reduce data provider cost.

Features

Distributed Web Sockets

Separate group of processes exposed over an API.
Go Lang is used to fetch real time prices over web sockets.
Number of workers can be specified. For example you can ask it to get real time prices of 100 scripts using 3 workers. It would evenly split the load across 3 workers.
Can be controlled using a REST API. New subscriptions can be added, existing instruments can be stopped without restarting the application.
No disruptions because a restart isn’t needed.

Run Everywhere, Control from Anywhere

One can register any number of computers (laptops, EC2 instances, Docker containers) by running a small custom processmanager on it.

Quick Statistics of All Compute Running Algos

It can then be further expanded to get some more system statistics. You can also submit jobs/algos to execute on them.

Start and Stop Individual Tasks on Each Computer

This makes it really convinient to start and stop algos without having to SSH into servers, logging into your laptop. One can login from anywhere and keep track of various on-going tasks.

Centralised Risk Management

Risk management should be centralised. The various algos running should communicate with each other and take informed decisions.
One can observe max risk change everyday and spot trends.

Money at Risk by Each Algo Segregated by Stock

Dashboard to tune risk and drawdown.

Unified Interface

Sensible intuitive APIs. One API for both real time and historical prices using real time workers. Makes writing programs a lot easier.
Can easily be expanded to support multiple data providers using bridges and adapters.
- Extendienble normalised data models for the same.

Ease of Access

Shortcuts on the Home Page to Perform Everyday Actions

Tech Stack

Languages Used: Go, Python, Node.js
Databases Used: Cassandra, PostgreSQL, MongoDB, Redis
Libraries and Frameworks
- Python: Flask, numpy, pandas, APScheduler, multiprocessing
- Node.js: React, Ant UI
- Go: Go Gin, KiteConnect, gabs
Quite a few places required intricate parallel or concurrent code with coordination using locks, semaphores and databases.
Data cleaning, finding the optimal update periods to reduce intervals of stale data and network cost.
Complex functional and object oriented paradigms for client algo classes.

Learning to Think in React by Building a Shopping Cart

Mon, 03 Aug 2020 00:00:00 GMT

I’ve noticed time and again that people don’t really understand the reason/history behind frameworks like React, Angular, Vue etc. It’s a different paradigm altogether (compared to Vanilla JavaScript or jQuery).

To really appreciate these tools, you’ll have to build complex user interactions and try keeping data and UI in sync using Vanilla JavaScript. You’ll quickly realise that apart from mutating data (shopping cart amount, product quantity), you’re forced to write boilerplate to change the HTML (store references to IDs and change .innerText). You are also forced to iterate and find documents within deep nested structures. Things are already complex and now we add API calls with error handling, loading spinners, notifications and modals to the mix.

Let’s build a shopping cart. First using Vanilla JavaScript and then the same cart using React 😊 My objective is to help you appreciate these tools, the pain points they solve, how they help with re-usability and learning to think in their paradigms.

Here’s the video of me coding and explaining the difference between both the approaches. Feel free to post any questions or comments.

YouTube Link

30 Days Road Map - Budding Front End Developer to your First Job

Wed, 06 May 2020 00:00:00 GMT

A handy guide to take you through the journey from a budding front end developer to get ready for your first job. You should read this guide if you already know the basics of front end development and are keen on bridging the gap between you and your first job as a junior developer.

Index

What do they Look for?
About Me
Bigger Picture (Essential)
- Build, Build and Build
- Decent Portfolio
Plan
Beyond Day 30
Credits

What do they Look for?

While interviewing for a junior front end developer / internship, your interviewer won’t expect you to know everything. They won’t be expecting depth of knowledge. The most important qualities that they would be looking for in an ideal employee are as follows.

Ability and confidence in building simple things.
Ability to break bigger problems into smaller problems.
Google and find solutions to these smaller problems.
Willingness to work hard and eagerness to learn.
Know how to start projects and continue making progress in chunks.
Many product based companies expect you to be familiar with Data Structures and Algorithms. I will not be touching upon them in this post.

About Me

I think I have reasonable experience to back the claims above.

I’ve interviewed at a few companies and by many potential freelancing clients.
Have been involved in recruitment processes.
Have interviewed candidates as well.
Handled classes for nearly 100 students, collecting feedback and clarifying their doubts on a regular basis.
Worked on a Resume Parser at my previous company. Was in constant touch with recruiting teams and I now think I have a fair idea of what they look for.

Bigger Picture

Now that you know what interviewers look for, let’s get right to business. I’ll explain some strategies to tackle some of those pain points.

Build, Build and Build

The main idea is to build numerous small to decent size projects. Reading about a concept (theory) and actually implementing it are two different things. You get those aha moments (when you really start understanding something) only when you actually code.

Additionally in the process of building you’ll learn new concepts. Since these are videos you can hear the video maker explain why certain things are done the way they are. These reasonings can be used to explain concepts when you are questioned by interviewers.

Every new concept you encounter read a little theory about it and then continue coding. You must strive to understand around 70% of the all code you write while following the videos/articles. If you don’t understand something, spend sometime on it. Don’t waste too much time on it. Make a note and move on.

Most of the things that you don’t understand will become clear after you are done with a few more items in the list. As you keep progressing your understanding also increases. I often continue even when I don’t understand something, I just make note of it and keep coding/copying from the video regardless. After a few videos or 1 mini project almost everything becomes clear.

This is the reason why I recommend that you keep coding rather than spending loads of time and energy everytime you don’t understand something.

Each item in the plan more or less tends to build on the previous item. So please avoid skipping.

Decent Portfolio

Save all the exercises and code you write. Organise them into neat folders. You will be uploading them to GitHub.

The following images demonstrates what I normally do. You can dive into the details here.

I have a folder called Practice and it is further sub divided into folders like JavaScript, Python, C and C++ etc

This is how an individual folder JavaScript would look like.

Many employers want to have a look at your past work or projects that you’ve been involved in. GitHub is an amazing place for that.

You probably haven’t worked before you won’t have prior work experience to talk about. You’ll be talking about the tutorials you’ve gone through, things you’ve built and show them all the hard work you’ve put by showing them your GitHub profile. It also signals that you are serious and excited when it comes to learning new things. You will have a lot more things to talk about and demonstrate.

I’ve noticed that longer interviews generally indicate higher chances of you getting the job. By having more things to show and talk about you increase your odds of getting the job 🙂

For instance I’ve been coding everyday at my job and this is how my GitHub profile looks. It clearly indicates that I’ve written a lot of code. When I tell the interviewer that I’ve written a lot of code and show them this, most of them are impressed.

Plan

Day 1 - 7

Important Note :- Keep all your code well organised. Don’t delete any of it. You’ll be pushing it online to show a decent portfolio.

Prerequisites

If you aren’t aware of the following topics, will take you a little longer. Please bear that in mind 😅

To follow along Days 1 - 7 you should have basic knowledge of HTML, CSS, linking stylesheets etc.

Goals

Get you more comfortable with HTML and CSS. Get to a state where you can build real stuff more easily. Also get familiar with mobile responsiveness without any framework. This should also give you a better about how CSS frameworks work in the future weeks.

Plan

Semantic HTML5
A lot of tutorials use Semantic HTML5 so it’ll be much easier if you know what semantic HTML is. Also it’s the recommended way of doing things.
- HTML5 Semantics (14 Minute Video)
- FreeCodeCamp Explanation with Images
Minimalistic UI - Responsive Profile Card with CSS (10 Minute Video)
Check video description for project files.
CSS Selectors
- Many beginners are not aware of common CSS Selectors. If you want to really understand other people’s code and get things done faster you should at least cover the selectors listed below. You may not use these selectors but will come across them in other people’s code.
- CSS Selectors 1 - 5
- CSS Selectors 6 - 10
UI Profile Card Using only HTML and CSS (10 Minute Video)
Starter files and code for the above can be downloaded here.
Media Queries in 5 Minutes (4 Minute Video)
Media queries are needed for mobile responsive websites. Quick explanation of what they are.
Amazing Transparent Login Form Just by Using HTML & CSS (6 Minute Video)
Please check video description if you want to download the code.
Awesome CSS Search Box (5 Minute Video)
Full Screen Landing Page - Responsive HTML5/CSS3 Tutorial (25 Minute Video)
Photo in the video is no longer available, use another picture of your choice.
CSS Flexbox in 20 Minutes (20 Minute)
Flexbox is a CSS Display Property that’s being used everywhere these days. It makes it really easy to write responsive websites. I feel it’s important that you start using Flexbox if you aren’t already. Many modern CSS frameworks now use Flexbox by default. I’ve noticed that many beginners aren’t aware of Flexbox. Please don’t skip this.
Code and live preview here.
Build a Responsive, Mobile First Website - HTML5 & CSS3 (45 Minute Video)
Code and live preview here.
Responsive Landing Page for a Website (60 Minute Video)
Check video description for downloading the code.

Day 8 - 14

Prerequisites

If you aren’t aware of the following topics, will take you a little longer. Please bear that in mind 😅

To follow along from Days 8 - 14 you should know Bootstrap fundamentals like the grid system, some basic Bootstrap classes.

Goals

Build more complex things using HTML and CSS. You will also learn the benefits of using CSS frameworks. We’ll be focusing explicitly on Bootstrap. The concepts are similar across many frameworks though.

Plan

Pluralsight Login Page Clone - HTML & CSS (40 Minute Video)
Code and live preview here.
Bootstrap Responsive Price Table (12 Minute Video)
Responsive Bootstrap Website Start To Finish with Bootstrap 4, HTML5 & CSS3 (1 Hour 25 Minute Video)
Download code from video description
Complete Responsive Website using HTML/CSS/JavaScript (2 hours)
This is quite long. You can always watch it at 1.25x speed to move faster 😉
Transparent Login Form with HTML5, CSS3 & Bootstrap 4 (25 Minute Video)

Day 15 - 21

Prerequisites

JavaScript basics like variables, loops, event listeners, functions, data structures and basic operations on them, linking scripts to a HTML page etc. You should also be aware of the DOM and perform basic manipulations like change CSS of an element, change text and so on.

Goals

Be in a position to make your web pages interactive (animations, manipulate DOM on events like clicks) and add basic logic.

Plan

Bulb On and Off using JavaScript (7 Minute Video)
Very simple. Just to get you warmed up.
Simple Calculator Using JavaScript (7 Minute Video)
Random Number Guessing Game (15 Minute Video)
A Simple Filterable List (17 Minute Video)
Something similar to a simple search form.
Stone, Paper, Scissors using HTML, CSS and JavaScript (1 Hour 25 Minute Video)
Memory Card Game (34 Minute Video)

Day 22 - 28

Prerequisites

Should have a feel for React’s philosophy, should know about Node.js, npm, know how to create a React.js project, basics like State and Props. If you don’t please watch React.js Full Course by FreeCodeCamp.

You will also need to know a little about external APIs. Most common meaning of an API - It’s a mechanism by which you can get data from an external source such as a server.

Goals

Become comfortable using React. Learn to write logic and also get a head start in structuring React applications. Be warned that this week will be more strenuous compared to the previous weeks.

Plan

Plan is to build a few simple projects, understand Higher Order Components, learn about React Router and understand Global Data patterns using Context and also a bit of Hooks.

I would suggest beginners not to spend time on Redux. It takes considerable time to learn. Hooks and Context come by default and are gaining traction. You get more return for the time spent by learning them instead.

React Component Lifecycle (25 Minutes)
An important topic. Time and again you’ll find yourself writing these methods. You’ll definitely encounter them in every medium size project at work. These same concepts can be applied while building Android apps, iOS apps, games etc.
Really Small Projects for Beginners
This will help you become more familiar. Choose 3-4 from the list and type them out. Don’t do everything as we are short on time :)
Recipe App using React Router (1 Hour 30 Minutes)
Get familiar with using External APIs and build an app using React and React Router. React Router is used when your application has more than one page. Most apps have multiple pages which makes it imperative to understand Router.
Protected Routes using React Router (15 Minutes)
A very common pattern. By writing a Higher Order Component you can ensure certain pages are only visible to loggedin users. The same Higher Order Component can be reused for multiple pages.
Why use Context? (16 Minutes)
A brief video explaining you the problem React Context tries solving. Once you know the problem it solves, things become more fun and meaningful. You DON’T have to install alpha version of React like he does in the video. The video is a year old. Context will directly work in your setup.
React Context & Hook by Net Ninja (2 Hours 40 Minutes)
A YouTube playlist to learn about React Context and Hooks.
Build a Clone of Your Choice
This is important. Do something that connects with you. Even better if you build more. This search term happens to point to some great videos. While learning from a tutorial you should try understand most of the code. You may not understand bits and pieces. Don’t worry about that. Don’t waste too much time trying to understand EVERYTHING. Keep moving forward and build new things.
6 Fun React Projects & Your Own React Project Ideas
For React Project ideas in case you’re still thinking about what to build.

Day 29 & 30

Plan

If you haven’t already pushed your practice code to GitHub start doing it now.
You’ve learnt a lot in the last 4 weeks. Revision is important. Trust me. I’ve been teaching for a while now and students forget a lot.
Quickly go through all these projects. You might be questioned on them. For instance
- Tell me about a recent project or tutorial that you worked on.
- The challenge you faced while solving a JavaScript bug and how did you finally solve it.
- What features of React have you explored?

Beyond Day 30

I have made a separate section for these because they are

A little time consuming or
There’s a chance that you might not appreciate the insights they offer as you wouldn’t have faced the problems the authors highlight or
Going deep into a particular topic might really not be necessary.

Advanced List

Every Layout - Really Good for CSS (Advanced. Free version is more than enough 🙂)
How to structure things the right way when using CSS. Really good. Must warn you though. It’s advanced and to really understand you should have written a lot of CSS. If you’re new to CSS please avoid this for now.
Build 30 Mini Applications Using Pure JavaScript
Fun, fast and informative.
Flexbox with Real World Applications
Again by Wes Bos. Introduces Flexbox but doesn’t stop there. You also build a couple of stuff using Flexbox.
Dave Ceddia’s Blog
Wonderful place for all things React.

Credits

Managing Processes with Supervisor - In Depth Tutorial

Tue, 05 May 2020 00:00:00 GMT

In this post I’ll show you how to setup Supervisor on Linux and become familiar with a few handy options and settings. If you haven’t used Supervisor in the past this post should give you a decent start.

Supervisor is a Python program that makes managing other programs (processes) a breeze. It can ease the process of starting, killing and restarting applications that you develop. I’ll be explaining various aspects that you need to keep in mind while using Supervisor.

Basics

Supervisor can help you administrate and automate the process of starting applications on your web server. It can run the steps specified by you and start multiple applications. Maybe you want somebody to automatically start the application after you’ve restarted your server. Supervisor can do all that and more.
Supervisor starts your application or script as it’s own child process. This way it can kill them when you want to. It daemonizes something that runs infinitely say a web server. You should not use it to control a process which has a definite end (say a task which takes 3 minutes and exits after that).
The command to start a particular process and various options are given via a .conf file.
Supervisor looks for files ending with .conf in certain directories. It then follows the instructions that you’ve given in those files.
Once a proper .conf file is written, starting any program is as simple as
supervisorctl start appname.
It daemonizes the process that you want to run. You can save logs that your application spits out in files.
Say your server runs 50 processes. It would be a pain to start every one of them whenever you restart the server. Supervisor has an autostart option which will start the process when Supervisor is initialised.
You could make Supervisor run a program as a particular user. There are loads of such options that can help make your day to day life easier.

Installing Supervisor

On Ubuntu or debian you can
```
sudo apt-get install supervisor
```
You can also install it using pip. Supervisor on Pypi.

Writing conf Files

Supervisor can be informed of any program that you wish to run via a .conf file.

I found this example from Digital Ocean to be very helpful. So I’m adapting a similar approach but with a different script. For the sake of this example and brevity we’ll write a small script that displays the amount of RAM that the machine sports.

Go to your home directory (~/) and save the following as displayram.sh

#!/bin/bash

counter=0
while true
do
    ((counter++))
    echo "Count is now $counter."
    cat /proc/meminfo | grep MemTotal
    sleep 1
done

Do note this is a never ending process. It runs forever because of the while loop. I repeat Supervisor can only manage and monitor never ending processes.

Make the file you just created executable.

chmod +x displayram.sh

Save the following as displayram.conf. Supervisor will use this file to control the process. Substitute abhirath with your username.

[program:displayram]
command=/home/abhirath/displayram.sh
autostart=false
autorestart=false
stderr_logfile=NONE
stdout_logfile=/home/abhirath/displayram.out.log

Move the file that you just created to /etc/supervisor/conf.d/displayram.conf. This is the default directory that Supervisor looks for programs.

Then on the terminal

supervisorctl reread

followed by

supervisorctl update

Finally run

supervisorctl start displayram

Our script starts instantly. You can verify that it works by opening /home/yourusername/displayram.out.log. It should be filled with lines displaying the RAM installed on the machine.

A better way would be to run the following on the terminal.

tail -f /home/yourusername/displayram.out.log
# or
tail -f ~/displayram.out.log

It will keep printing out new entries on the terminal.

You will also need to familiarise yourself to spawn more than one process - An example which requires running multiple processes to start an app.

To stop the script we just started run

supervisorctl stop displayram

A Few Options Explained

Command

The way via which Supervisor starts a particular program. Give the command / path that you normally use to start your program.
Commands are not equivalent to bash or your typical terminal commands by default. They are fairly similar but a command that you use on your terminal may not yield the same effect here.
Commands can contain absolute paths or relative paths. The displayram example uses absolute paths. If you want to use relative paths you need to set the path. You can read more about it here.
You can ensure that your command is treated as a bash command by using bash -c "command here". Our earlier example can be rewritten as

[program=displayram]
directory=/home/abhirath
command=bash -c "./displayram"
autostart=false
autorestart=false
stderr_logfile=NONE
stdout_logfile=/home/abhirath/displayram.out.log

Stop and Kill as Group

Sometimes your program doesn’t really get killed. They might have children that they don’t kill. So to actually stop them Supervisor has to be mean and kill every process that might have spawned. One possible reason is that they expect a different kill signal.
You don’t have to use them unless you notice that your program is still running even after you explicitly ran supervisorctl stop programname
This helped me a lot - Really killing a program.

Logs

stderr_logfile - the file to which it shall redirect stderr to. If set to NONE it will not log output from stderr.
stdout_logfile - the file to which it redirect stdout to. This can also be set to NONE.
stderr_logfile_maxbytes - maximum space that the log file will take before Supervisor uses a new file to store logs. You can input a numerical followed by GB, KB etc. By default it’s set to 50MB.
stdout_logfile_maxbytes - same as above but for stdout.
stdout_logfile_backups - the number of backup files it stores. Say you’ve set it to 3. After the first files exceeds the value of stdout_logfile_maxbytes it creates a second file then a third. If the third file exceeds the size it proceeds to delete the first file and continue logging. In all it would use a maximum of 3 * 50MB in our case. By default it has a value of 10.
stderr_logfile_backups - same as above but for stderr.
An example which uses a variety of log settings.

Supervisor saves it’s own logs as well. They are different from the logs you set for your program. These logs contain information about Supervisor. They are stored in /var/log/supervisor/supervisord.log To see the last few logs you can tail /var/log/supervisor/supervisord.log

A useful read - Options available under program.

Some Common Errors

1. You get a message similar to unix:///var/run...sock no such file?

It means Supervisor isn’t running. You will have to start Supervisor. sudo service supervisor restart

2. Spawn errors. Supervisord is unable to start your program?

Check the error log and output logs. Supervisor keeps appending stuff to your program’s log file. So it’s a good idea to use tail -f /path/to/program/log/file to keep track of changes happening.
Check the environment and directory options under your program. Give absolute paths like /home/abhirath/so/and/so.

3. Supervisor doesn’t start automatically after bootup?

On Ubuntu 16 you can enable it by sudo systemctl enable supervisor. Also read this stackexchange question. - init scripts for various operating systems.
Frequently new .conf files that you write are not recognised. If that is the case do the following:- Check the syntax of everything in the file. If there’s an error Supervisor doesn’t even show the file. Also the files must have an extension of .conf.

4. Any signs of the process / app that you want to start is already running?

They might already be running. Check the status using supervisorctl appname status.
I’ve seen .NET apps that don’t kill their child processes by themselves. So even after you stop a program their child process might be running. For instance you might get errors like port is already in use.
Read about stopasgroup, killasgroup and stopsignal over here
stopasgroup, killasgroup work on individual programs. They are not to be confused with group applications
Really killing a program.

Vuvuzela - A Private Messaging System

Fri, 22 Mar 2019 00:00:00 GMT

I gave a talk about Vuvuzela at Papers We Love, Chennai. You can find the slides here. This blog post tries to cover the points we discussed orally and therefore aren’t a part of the slides.

Paper:- Vuvuzela
Recommended By:- Ganesh Srinivas

Metadata

In the context of a call things like who did you call, location, duration of the call, the number of times you called up the person etc. Anything that describes additional parameters about the content (recording of the phone call in this case) can be termed as metadata.
Verizon Injects Custom Headers to Un-Encrypted Traffic is an interesting read. It doesn’t directly address metadata but custom headers definitely fall under metadata. The article will give you some insight as how to such info can be used.
State of the art privacy mechanisms even go to the extent of hiding your screen size and resolution as it can be used to identify a whistleblower, journalist etc amongst a pool of their suspects. Ganesh mentioned about an interesting flaw in browsers which goes by the name Canvas Fingerprinting.

Existing Work

Some systems offer strong privacy guarantees but do not scale. Some systems broadcast every message to every user. Other systems use Private Information Retrieval (PIR) to hide metadata. Simplest PIR technique is to send the entire database to every user who requests for information. This way observing parties (using Man in the Middle Attacks for instance) will not get to know which particular information (or message from a recipient) they were trying to access.
Systems that scale do not hide metadata and a persistent adversary can easily correlate activities with a particular individual. Tor and Mixnets are such systems. A powerful adversary can manipulate traffic and several nodes. Say amongst thousand users he stops Charlie from accessing a website and the same very same instance the adversary notices that the particular website stops receiving traffic. My manipulating and making the right moves over a period of time the identity of the user can come to light.
I like to think of it this way. Tor plays with space alone (the bytes that you send across the network go through various other servers thus they aren’t where they are supposed to be) and mixnets play with both space and time (adds delays and shuffles the request through various servers as well). Thinking in terms of space and time helps clarify a lot of concepts as they comprise the state which we exist or can manipulate.
Vuvuzela uses two protocols to operate. One protocol to initiate conversations and another one to exchange messages between parties who’ve already established connection (by sharing keys i.e).

The authors expresses it’s privacy guarantees in terms on differential privacy. I don’t really understand them in depth but basically the adversary can never be sure. His suspicions on a person can increase with time but he can never be sure if he was involved in the communication or not as to the external observing party everything looks the same.
Vuvuzela uses a lot of network bandwidth to operate. Also it stores all the incoming and outgoing messages in volatile memory. Additionally you would also need multiple servers following the Vuvuzela to add enough noise or cover traffic. So hardware costs definitely lean on the higher side.

Todo

Explain both the protocols
The small additional things (like client, uniform padding, participate in every round, rate of messaging etc)
Algorithmic complexity
How different variables or state of the server is hidden

Interesting Stuff Mentioned in the Meetup

Differential Privacy using Coin Tosses (Differential Privacy by Sampling)
How Mozilla Anonymises the Information it Collects (Prio). You can read a little about it here.

Learning to Think in React by Building a Stopwatch

Thu, 14 Feb 2019 00:00:00 GMT

Here’s the video of me coding it. Feel free to post any questions or comments :)

Onion Router and Mix Networks

Thu, 17 Jan 2019 00:00:00 GMT

Index

Introduction
Prerequisites
Onion Router
Mix Networks
Security and Meta Data Abuse
Scaling

Introduction

I wrote this article as supplementary reading for a talk on Vuvuzela. I hope it can be read as a stand-alone post as well but that isn’t the goal.

In order to understand Vuvuzela I feel it’s important to understand the limitations that the earlier mechanisms suffer from. This is a short post contrasting Onion Routers and Mix Networks. It is important to note that both these protocols fail to hide metadata. I’ll reiterate and give some more context to drive this point home, just knowing that they don’t hide metadata is enough for now. I’ve also linked to articles I’ve read in the past and also the ones I’ve stumbled upon whilst reading the said paper.

Prerequisites

You need to have a decent idea about the following concepts.

Public / Private Keys and Digital Signatures. This and this provide a decent explanation as to how keys work in general.
Proxy Servers.

Onion Router

A detailed explanation of the protocol and workings can be found here.
Use it under the assumption that the adversary cannot monitor all nodes. If you can protect just one server in the chain, you can be assured that the content of your request and response stay private. Under normal circumstances either source or destination or both remain hidden. If the first server is compromised, the source of the request (the client trying to visit say YouTube.com) is given away but the end destination YouTube.com stays hidden as everything in encrypted using multiple keys. Similarly if the last server is compromised the attacker will get to know the end destination (i.e YouTube.com) but will not know where it originated from.
The chain of servers chosen normally spans across diverse geographical boundaries. This makes it harder for perpetrators to manipulate network traffic as rules, regulations and privileges vary across political divisions.
Unlike mix networks, onion routers can be used for time sensitive data as well. Requests are forwarded immediately. Mix networks normally wait for quite sometime before they start forwardings requests. They are safer but the message takes longer to reach the final destination.
The client (i.e the user who’s using the Onion Router protocol to protect his privacy) needs public keys of all the servers in the chain. However each server can only decrypt it’s layer and the content stays hidden. Each server just takes it a step forward towards decryption but cannot actually see the content.
If the website visited by the user uses HTTPS, even the last server in the Onion chain cannot see the content even after decrypting all the layers. The last server will know the end destination of the request but it will not know the content of the request nor will it know the source of the request. It just knows that the second last server in the chain has sent it some encrypted data. The last server on decrypting the data, will find details of the destination server (say YouTube.com) and whom it has to forward the response to (the second last server, Onion Routers follow the same chain of servers).

Mix Networks

Please read this. It explains it pretty well. It also draws real life parallels to an observer in a crowd. It really clears things up.
You could watch this video if you have 12 minutes to spare. Not required though.
It provides an additional layer of protection. The content of the requests are protected even if the adversary monitors all the nodes in the network.
Does not depend on route reproducibility.
It cannot be used for time sensitive data.
It too is prone to meta data analysis.
I like to think of it this way. Tor plays with space alone (the bytes that you send across the network go through various other servers thus they aren’t where they are supposed to be) and mixnets play with both space and time (adds delays and shuffles the request through various servers as well). Thinking in terms of space and time helps clarify a lot of concepts as they comprise the state which we exist or can manipulate.

Security and Meta Data Abuse

Adversaries are very powerful and have great reach. They can control ISPs, cloud providers, assign people to monitor suspects/victims over very long durations of time etc. They therefore can block certain users, observe traffic flow patterns from a user to the server, between servers and so on.
It is fairly easy to know that the victim / suspect is connected to such a server (Tor, Mix Net etc) as they are generally publicly known, basic traffic observation will reveal that. It is not trivial to track his activity and prove that he did something particular. That’s key.
A patient adversary can easily follow patterns in case of Onion Routing. Say Alice visits abcxyz.com everyday at 8 PM, an adversary can block all other users other than Alice. He can then look for activity, he will notice that Alice’s network is active and the request flows through a particular chain. Since she’ll be the only person using the network it’s quite evident that she’s the one visiting the particular website.
An adversary can monitor the network and identify individuals by looking at request sizes. Say Alice, Bob and Charlie send a request to abcxyz.com or are engaged in a chat. Network traffic can be monitored at the entry and the last server. Alice sends abnormally large data compared to Bob and Charlie, so the adversary can find out the recipient of the request (abcxyz.com or chat server) from the last server in mix and correlate it with the sizes of the request sent by Alice, Bob and Charlie.
Mix nets make it harder to pin requests to a particular user but a patient adversary over many rounds can statistically correlate information to a user.
Adversaries can disconnect everyone except two parties and try figuring out if they are chatting or not by intercepting their traffic. If request request by Alice corresponds to a request to Bob, it isn’t far fetched to assume that they are chatting.

Scaling

Both these approaches are very scalable. A simple strategy would be to add more servers following the respective protocols and the clients are given an option to choose the chain of servers instead of using all of them (based on the load of each server). The client would then make appropriate requests with the given data. Say you are very paranoid in general and therefore choose a longer chain not limited by the boundaries of a single country. Since requests of a client does not have to go through all the servers, both these approaches are easily scalable. To the best of my knowledge and understanding both scale linearly.

GD50 Lecture 05 - Super Mario Bros

Sun, 13 May 2018 00:00:00 GMT

This is a part of a series of notes. You can find notes to other lectures here Please feel free to bring to my attention mistakes, other interesting resources and feedback via the comments section. I’m all ears and will do my best to incorporate them into the blog post.

Resources

Continue Statements in Lua - Read this answer. Other options are decent as well.
Factory Patterns - This site explains widely used programming patterns really well. Also has code samples.
Entity Component Systems Explained - A pretty comprehensive explanation with examples. It’s a part of an amazing book.

Intro

Not using the original sprite sheet due to copyright issues.
Released in 1985, it kind of revived the gaming industry after the crash in the 70’s. Probably the most famous game of all time.
2D Platformer. Quite complex for it’s time, move around, effected by gravity, spawn enemies etc.

What Goes Into Mario?

Tilemaps - Each level is designed using individual 16 x 16 tiles.
2D Animations - Kind of like a flip book.
Collision Detection - We’ll be deviating from AABB that we normally use. We don’t need to iterate through every tile in the world checking for collisions.
Camera - We track the player and move the world along with him. The notion of a camera is introduced.

Some columns in the level there are no tiles (chasms the player can fall into), some columns are a bit higher (player has to jump), some have power ups and so on. All these variations in pattern put together create a level.

Tilemaps

2D array of numbers. Each number normally depicts the kind of tile that should appear on the corresponding spot of the level. They can have a host of other properties like allow to pass things, trigger an animation and so on. You could do this by associating properties (using Tables in Lua) against each tile. The sky is the limit.

tiles0 “Static Tiles”

Colour in the background and just render a few tiles. That’s all.

Question by Colton:- What would be the first step to implement this?

Student Answer:- Loop over tiles and render them.

Colton goes on to explain that we’ll create a table to store tiles. We just won’t store numbers. We will store each tile as a mini table that will have an id as well as some properties (sort of like metadata).

Create a sprite sheet using generateQuads.
A small sprite is used. We index into a the table of quads just generated to get the relevant sprite.
Line 39 - 42 => Generate a random colour. It’ll denote the sky.
Variables mapWidth and mapHeight are used to to fill a table with appropriate number of entries for this demo.
TILE_SIZE is a constant set to 16. It’ll be 16 throughout the lecture.
Lines 47 - 50 => Populate the table with data denoting the structure of the map. If height is less than 5 (top portion that is) make it a sky tile else a ground tile. It’s just a table with integers denoting sky or ground. We do not store the actual quad.

Line 74 - 85 => Iterate over and use the above generated table to render. Sky tiles have no quad. If it’s a ground tile render the appropriate quad.

Final outcome:- Screen with tiles and a random sky colour.

tiles1 “Scrolling Tiles”

Types of scrolling in games.

In some games you cannot scrolling. For example in Temple Run you cannot go back and forth. The game keeps moving forward all the time.
Then there are games like Mario where you make the entire world move respective to the player.

love.graphics.translate() Moves the coordinate system. Everything drawn to the screen after that gets rendered respective to the new coordinate system. So we shift everything and move the world. This gives the effect of the world moving as the player moves. We pass in X & Y based on the player position.

Demo => Key presses now move the tiles. If you move too much you will reach a point where there are no tiles. Normally such information is hidden from the user (recall that games are illusions).

Line 95 => Translate by negative camera speed. If the player moves to the left, we want everything to move to the right to give the illusion of the player moving. Y coordinate is set to 0 because we don’t want the map to move vertically. Math.floor is used as decimal values might cause blurring (called Blur Artifacting). Any time you condense (texture rendered on a small virtual window) or magnifying values while using a virtual screen there are chances of weird blurs appearing.
Line 80 - 87 => Take care of updating camera scroll based on right and left key inputs.

We translate before we draw. Everything drawn after the translate gets effected by it.

Love 2D is a low level game engine which gives more control and is good for learning. It has no camera objects and the likes. Higher level game engines like Phaser etc have camera objects which encapsulate this translate behvaiour.

character0 “The Stationary Hero”

We again generate quads from the sprite sheet and just draw the first quad. It’s just static and does nothing. The other quads you notice in the image files are going to be used for animating the character.

character1 “The Moving Hero”

Now instead of camera scroll, have a variable called characterSpeed. In this demo the character moves but the world (i.e tiles) are stationary.

3 things yet to be done.

World moves as character moves.
Animate character as he moves.
Clamp (or limit beyond a certain point) if the character reaches the edge of the world. In such a scenario, the character moves but the world doesn’t.

character2 “The Tracked Hero”

The camera is fixed on the player. World moves relative to the player.

This demo is just to show how the world changes with respect to the player so does not account the case where the player reaches the edge. See the actual code in mario/src for that.

character2/main.lua Lines 93 - 103 =>

function love.update(dt)
    -- update camera scroll based on user input
    if love.keyboard.isDown('left') then
        characterX = characterX - CHARACTER_MOVE_SPEED * dt
    elseif love.keyboard.isDown('right') then
        characterX = characterX + CHARACTER_MOVE_SPEED * dt
    end

    -- set the camera's left edge to half the screen to the left of the player's center
    cameraScroll = characterX - (VIRTUAL_WIDTH / 2) + (CHARACTER_WIDTH / 2)
end

In the line cameraScroll = characterX - (VIRTUAL_WIDTH / 2) + (CHARACTER_WIDTH / 2) character width is added so that the world’s coordinates line up perfectly in the center. Also notice that we check for keypresses and keep updating characterX.

In this example we just translate the world by the X axis and leave Y unchanged. Say mario could jump extremely high or we were to implement some sort of vertical scrolling game, we would have to translate on the Y axis as well (using the same approach as given above).

character3 “The Animated Hero”

If the player moves loop through some image to create the illusion of animation. Animations are nothing but a series of images shown in succession.

Created a class called Animation.

Pass in the frames you want animated.
Time interval between each frame (quad) switch.
If all frames have been looped through, repeat from start again.

The class also includes logic for animating single and multiple frames. Idle animation for instance has a single frame. So by writing it this way the same class can be used for all animations.

What about moving left animation?

We just flip the drawing when calling love.graphics.draw(). If you recall it takes in optional arguments to flip it (rather scale, rotate i.e translate it). So we have to keep track whether he’s walking left or right and flip it on the X axis if he’s walking towards the left.

Animation.lua => Notice that while initialising the class the current frame it set to 1 (line 17). Also on line 22 we check if there are more than one frames. If there’s only one frame it’s kept constant throughout otherwise on the completion of the specified timer we keep changing the frame.

Notice that we have a function function Animation:getCurrentFrame() that returns the frame. The main render function basically calls this function every frame to get which quad to render. So every few hundred milliseconds it changes internally and therefore returns the next quad and the main render function shows that quad.

The animation class does not actually render the quad on screen. It just keeps track of the ongoing frame in the animation. Any render function can use this object to query the current frame and render it.

main.lua Lines 51 - 58 => The idle animation object is defined here. An interval is passed into it for the sake of consistency even though it doesn’t make a difference. He’s also defined movingAnimation after that. Notice that the sprites 10 and 11 are the sprites which show the character moving (the walking away frames).

Lines 60 and 67 => currentAnimation keeps track of the animation currently activated. direction keeps track which direction the player is walking in so that we can flip conditionally flip the sprite while rendering.

Line 150 => love.graphics.draw() gets the current frame from the animation object by calling getCurrentFrame(). The animation object keeps updating the frame as time progresses and the render function gets the current frame to show from the animation object.

Lines 150 - 160 => All these lines are just one function call. When a sprite is flipped, it is done so with respect to it’s origin. Default origin is the top left corner. So by flipping the character it will drift a bit from the center. So while calling love.graphics.draw() the last argument we pass, we set the origin to be the center of the sprite. Normally the sprite is recentered when we flip sprites and want consistency.

If you’re wondering how does the current animation switch?

Lines 115 - 125 => Continuously checks for keypresses. If right or left key is pressed, change currentAnimation to movingAnimation else let it remain idleAnimation.

character4 “The Jumping Hero”

Question:- How to jump?

Check for space, we need gravity, delta y. (Recall Flappy Bird and how we tracked gravity and delta y).
Take care of jump animation.

On closer observation it becomes apparent that we need two states. One while going up and the other to indicate that we are coming down.

Why do we need two states?

While jumping we destroy blocks, when falling down we kill enemies. The inverse doesn’t hold true. We need a way to differentiate.

The two states share the same animation but have different behaviour.

Main.lua 120 => if key == 'space' and characterDY == 0 then we ensure that DY is zero so that the player only jumps when he’s on ground. If he’s mid air, DY isn’t going to be zero and allowing him to jump in such a state will allow a player to jump infinitely in mid air.

Main.lua 132 - 135 => We haven’t reached collision detection yet so we quickly hack it up. If the character reaches below the ground set his dy to zero so that he stops moving.

Procedural Level Generation

Colton found the easiest way to think and reason about generating tiles for this game was to go column by column. First you start with no tiles (just the sky denoted by a background colour). Then you go column by column and ask do I want ground here, if yes fill it will tiles from the bottom edge till ground level. Do you want a raised pillar here, if yes add a few more tiles post ground level. If you don’t want ground in that column (i.e a chasm or a ditch) don’t fill any tiles at all in that column. All this can be done with boolean flags and math.random().

Try thinking of ways to generate it row by row (horizontal strips across the screen) and it will dawn upon you that it’s much harder. Column by column helps to think of it in individual units (this column has a chasm, this has normal ground and so on) but row by row you are forced to deal with tiles that aren’t a part of the same structure.

Taken from the book:- Code Complete 2nd Edition Chapter 2.1

The essence being how we see or model things (abstracting into classes or functions, think of a map as a 2D grid and those sort of stuff) makes a lot of difference while working on them. It opens up new possibilities, greatly simplifies and can even break stuff.

You could easily extend this idea. Say you want a pillar of varying width you could introduce a width variable and fill up the next few columns with a pillar and decrement the width variable by one on every iteration.

Placing objects (grass, enemies, powerups etc) can be dealt with after you are done with the iteration that places all tiles. You can conditionally place objects on certain types of tiles (should not place grass on ditches and those sort of things). You can also keep track of where you’ve placed obstructions and place the next obstruction at a reasonable distance from the last.

He then goes ahead to show how a varying sprite sheet with a simple algorithm like the one described above can be used to generate rich levels. Assortment of sprites for the ground tiles, topper (thin layer near the top most tile added for visual appeal) tiles, backgrounds, obstacles make it look vivid.

The sprite sheet is well organised into similar chunks. This organisation provides easy programmatic access (splitting the sprite sheet into quads and accessing different variations of the same element. Most importantly the same can be done for all of them).

level0 “Flat Levels”

Topper with ground. How it’s done?

He’s just taking the topper and the tile set (ground tiles) from different parts of the sprite sheet. We don’t fix a place for toppers so as to keep it general enough to work with pillars and chasms. There’s a flag which indicates whether a tile has a topper or not. If it’s set to true if it’s the topmost tile of that column.

Splitting tile set and toppers?

Have a look at tiles.png for a better idea of what we’re about to do.

level0/utils.lua => We need to write a four way nested loop to get the tiles and toppers. The first two levels of nesting is to actually get one of the 30 (3 X 10) different segments of tile sets. After that’s done, we’ll have to iterate again and get tiles across the X and Y dimensions again. So we’ll need a two level nested loop for that. Have a look at GenerateTileSet(). It takes in the actual quads, the X and Y of the large segment (3 columns by 10 rows) and the size of each segment as well. Basically taking a segment of the sprite sheet and extracting the desired group of related tiles from it.

This is only possible as the sprite sheet is structured and laid out in an organised fashion. Programmatic access would not have been possible if they were randomly spread out.

level0/main.lua =>

Constants for width, height etc.
Gets quads from the respective sprite sheet and then split them into relevant sets using GenerateTileSets().
Lines 223 - 249 => Use a table to represent the entire world. Fill in blank tiles to represent the sky, if height is equal to 7 give it a topper (thin layer that the player actually walks on).
Line 139 & 140 => The tile and topper set is a result to something random when r is pressed.

level1 “Pillared Levels”

How to put pillars?

Just put up tiles above ground level in that column.

You could have a flag spawn pillar which controls whether to have a pillar or not. You could also pass in the width or something and draw the same height multiple times and set the flag back to false.

main.lua Line 227 - 236 => Fill the entire map with the sky tile. So the level is basically empty at this point.

Once done we don’t have to worry about inserting or managing the table. We can just replace the stuff we want.

main.lua Line 239 - 259 => Go column by column to generate the level map. Randomly set a flag to generate a pillar or not. If the flag is true fill in positions 4, 5 & 6 with tiles (these are 3 positions above ground.) Also place a topper on the top most tile.

main.lua line 253 to 259 => Fills tiles to create the ground in that column. If a pillar exists don’t put a topper else place a topper as usual.

You can also create mario like steps instead of straight pillars in a similar manner. Keep reference to the previous height and thn iterate (each iteration reduce height). You could also keep track of width and keep increasing height till half way up and reduce height from there on.

Because of the code’s random nature it’s possible that two pillars spawn next to each other. Pseudo random generators can also surprise the person who wrote the code.

level2 “Chasmed Levels”

How to generate chasms?

You just skip a column. So at the start itself you randomly decide if you want a chasm in the block or not (in the code there’s an arbitrary set to math.random(7) for 1 in a 7 chance, which in general is a bad practice).

I think (I’m not sure myself) the reason being you have no control over where the chasm ends up (like the very first step where the player stands) and it probably is a very raw approach, we could do better by having a few hand spun rules like at so many intervals, only after a certain point in the level.

Lua has no continue statements, so we use a goto statement and avoid rendering ground in the column. Using ::continue:: is a community accepted standard work around for the same.

Audience Question => Level generation do we do it all at once?
In mario we do it all at once. It’s pretty small and can easily fit in memory. Also it’s much easier to teach like that.

Incremental approach can be taken for infinite runners. You split the game into parts. As the user approaches the right end, generate the new level, obstacle etc and append it to the level. You can discard the level or obstacle that the user has already crossed (in infinite runners you normally can only move in one direction).

Audience Question => Do we render the entire level at once?

Yes . Since it’s a small game rendering the entire map 60 times a second isn’t a problem. Large game render only a certain section. You would write your code to ensure it renders only areas near the camera or the user.

Related resource from a previous lecture Games are Illusions.

Tile Collision

We have a fixed 2D tile system. We can get to the tile from the x & y coordinates. We can use that to your advantage. We just have to check that particular tile. If you recall with AABB collision detection you have to iterate over all the tiles to check if it’s colliding with the player. In this case we know where the player is and we just can check that tile.

You can find the relevant function in mario/TileMap.lua. Also I’d recommend seeing slide#25 for more details.

Tile collisions while jumping up => Just check both the top edges of our character (which tile the head hits). Checking only one side will miss a collision if half his body is under a tile and the other half is not blocked by anything (and we happen to check the side which is not under anything). We just need the tile our player will hit. We get the index of the tile and check if it’s solid (property set via a flag in the table).

Notice how the top portions of the bounding box of the player touches the tiles (it therefore will trigger a collision even though the player’s head is not touching the box). Reduce one pixel from each side which checking for collisions so that it can go through gaps that are exactly of the same size without triggering a collision.

To get the tile index => 6th from top, 5th from top and so on , we just have to divide the player’s Y coordinate by a tile height and add one to it. The same things can be done for the X coordinate as well. We can then index into the tilemap table.

Bottom Collisions => We do something very similar for the bottom. This time we check for collisions with the character’s bottom edges. The coordinates of the bottom edges are (x, y + height) and (x + width, y + height). The same is done for checking right and left collisions.

Remember that top, right, bottom and left collisions are checked in different states. We need to check top collision while jumping for instance. We needn’t check for top collisions while walking towards the right for instance.

mario/src/TileMap.lua =>

Lines 27 - 33 => We can easily get the current position of the player and from that we can easily get the tile the player is on.
Lines 28 - 29 => If the player happens to be beyond the map boundaries then return nil rather than a tile.
Line 32 => Lua tables are one indexed but geometric coordinates are zero indexed. So we add 1 to the indices we get.

Advantages over AABB

Just find out which tile the player is on.
We don’t have to iterate over all the tiles and perform AABB collision on them (recall Breakout).
Kind of like a linked list vs array. Array you know exactly where to look and can get what you want immediately. Linked list you have to iterate one by one until you find it.

This is good to repeat again. You check for collisions at different points on the character based on the different actions he performs. Say he is jumping you check for collisions on the top left and right edges. Since the gaps are wide enough to let our player pass between them we only check the top edges while jumping. Assume that our player is 8 tiles tall, in that case we’d have to check for collisions across his entire body even while walking.

Game objects need a different treatment and are a little more complex. Have a table of some objects like snails and we have to iterate over them one by one to check for collisions. It’s because they have their own X & Y coordinates and unlike the tiles they aren’t fixed at a place.

Entities

Almost anything that you want can be an entity.
In this particular game Colton has modelled everything that moves or is sentient as an entity (snails for instance).
They can have their own state.
He explained a little about Entity Component Systems. In such systems everything is an entity, each entity has components (we’ll discuss it below). Unity for instance is an entity component system.
I like to think of them like CSS classes. Every new class added to the HTML element gives it additional properties.
Everything has properties and these properties ultimately define the behaviour of the object.
Instead of a deep tree with inheritance (Say you have a base class Monster, Goblin a subclass inherits from it, the subclass GoblinWarlord inherits from Goblin and so on), you have a container which you compose (or fill) with components or properties. All the pieces together create what you want rather than having to inherit from parent classes.
Entities are often objects that do things or perform some action. In this game the tile map are just statically rendered and are not entities. Stationary objects are not considered to be entities as well.

Interacting with Entities

mario/scr =>

GameLevel.lua Line 14 - 16 => References to entities (enemies), objects (the non living stuff like obstacles, coins etc) and tilemap.
GameLevel.lua Line 36 - 46 => Update (position, health, consumed or not etc) the tilemap, the objects and entities. The main update function calls this. Notice how the individual update function of each object and entity is called.
GameLevel.lua Line 51 - 58 => This piece of code in responsible for rendering them. It calls the individual render function of each object and entity.

Player Falling State

mario/src/states/entity/PlayerFallingState.lua Lines 62 - 78 => Iterate over every game object. If the game object is solid change dy (rate of change of height) to zero and move the player just above the object. If the object is consumable (gem for instance) remove the consumable object without doing anything to the player.

Depending on the object do different stuff. If it’s an enemy it dies or kills you, if power up show the relevant effect and so on.

Check resources for the link on Entity Component Systems.

Game Objects

Things that don’t go well with the tile dimensions, maybe they are of different sizes, appear at weird places etc. Examples are powerups, ladders, coins, ropes etc.
Test them using AABB collision.
We could have modelled them as entities but in this project it’s not the case.
In this game, entities are living sentient beings, game objects are non living. Colton adds that for small projects as this everything could be entities.
Definitely have a look at Slide #30.
In bigger games, different varieties of the same object exist (different types of bricks, gems etc). In such a case we could have a class for object and have sub classes which inherit from the base. This way we’d easily model a lot more objects.

How would you implement ladders?

Check for a collision with a ladder. If it’s true and the up arrow key is pressed then enter climb state. If reached the end enter walking state again.

You can generate a lot of objects based on functionality. For instance:-

Ropes and ladders all trigger a climbing state.
Bushes, mushrooms are cosmetic. Nothing happens when you jump on them.
Spikes and a few other objects are lethal. You can have a flag isLethal against the object and on collision it should kill the player.

LevelMaker.lua =>

It generates game objects randomly.
Every object has flags like collidable, consumable, solid etc. when a player collides with an object the respective callback function is activated. When a gem is consumed for instance, score increases, a sound is played and the gem disappears.
Each game object also has an associated texture, height, width, x and y coordinates.

mario/src/LevelMaker.lua =>

Lines 56 - 74 => There’s a one in 8 chance to spawn a pillar. There’s a one-eight chance of having a bush on a pillar. So in all there’s a one in 64 chance of having a pillar with a bush.
Line 85 - 91 => Look at how the properties are set. Especially note line 91 which sets collidable = false. Bushes therefore don’t trigger anything when collided against.
Line 97 - 112 => Randomly deciding to generate a block or not. On line 112 solid = true. The user cannot pass or move through them.

The above set properties are tested for and certain actions can then be triggered depending on these properties. Open mario/src/Player.lua and have a look at the methods Player:checkRightCollisions(dt), Player:checkObjectCollisions() to see how they are used.

Audience Question => Wouldn’t it be better to have a helper class that takes care of creating objects?

Such a pattern is called a Factory Pattern (or at least is very similar in spirit).

At this scale it isn’t necessary. So he just created one class called GameObject that abstracts away creation of all objects. He also felt the need to reduce the number of files to simplify the code and increase comprehension. If Colton were to design a large game he would probably create a class for each kind of object like a bush class, a gem class and so on. In a large game there’d be many kinds of gems, bushes and so on. The class would take optional parameters to create different variations of bushes and so on.

The current GameObject class is abstracted enough to allow for custom event handlers that can be tested for conditions. We can thus define custom behaviour for them. For instance a gem is consumable but a bush isn’t. So on collision we check if the flag consumable is true and then we call a custom function.

He again briefly hints at Entity Component Systems. It’s a pretty neat way of modelling stuff. You just add components like consumable to an object such as a gem.

Check resources and check out the link on Factory Patterns.

GD50 Lecture 04 - Match 3

Fri, 09 Mar 2018 00:00:00 GMT

Resources

What are First Class Object? - This is the link I referred to when I wanted to know more about Python function.
When to use Anonymous Functions?
Bubble Sort Visualised.
String Strip in Python.

Game Description

Grid of small objects that you should group into collections of 3 or more. The score is directly proportional to the number of objects you get together. Whenever you match similar object together they disappear and are replaced by more objects. Bejeweled and Candy Crush are some Match style games that you probably are aware of.

timer0 “The Simple Way”

How would you implement a simple timer (1 2 3 and so on)?
Some variable that you keep modifying by adding DT (time between each frame). You could also simply display it on the screen as well.

main.lua => Variables currentSecond and secondTimer are declared. secondTimer stores the time progression since the last tick and is reset once it crosses 1. Every time secondTimer is reset, currentSecond is incremented.

Note that we don’t set secondTimer to 0. We instead do secondTimer = secondTimer % 1 so that if secondTimer is a little above a second, it keeps that value. The excess value reduces the span of the next tick (Check this out for more details). Lines 40 - 47 implement the described functionality.

You can expect every DT to be approximately equal to 1/60^th of a second as the screen gets updated 60 times a second.

This is a bad way of doing it. It doesn’t scale.

Also keep in mind that Lua doesn’t differentiate between a float and an int. Lua only has one numerical type. So you can use modulo 1 on a float as well.

timer1 “Also The Ugly Way”

5 timers on the screen this time. 5 times the code as the previous example. He mentioned in the last section that it isn’t scalable. He shows it here. Since there are 5 timers he’s used a loop.

timer2 “The Clean Way”

Global time object which manages everything with the power of anonymous functions. Please note that we could use normal functions as well. The task at hand is very trivial and we don’t use it in more than one place. Hence Colton has resorted to anonymous functions.

We use a library called Timer (it’s a part of the knife ecosystem).

-- interval is in seconds
Timer.every(interval, callback)

Timer.after(interval, callback)

Timer.every(interval, callback) is similar to JavaScript’s setInterval. You can specify a function to be executed every x seconds forever. An anonymous function is passed (you can pass in any function, it doesn’t have to be anonymous). The function gets called everytime the interval runs out.

Line 21 - 32 => Declared the intervals and second counters in a table. Iterate over those values and just set a Timer. Notice the nameless function on line number 29.

Notice how the code falls from 98 lines in the earlier example to 70 lines. We can even specify more timers without increasing the lines of code (just add more timers to the table). This is very scalable. It’s declarative as well (just specify how long you want a timer. No additional logic).

Timer.after(interval, callback) does something once after the given interval.

You should checkout this library called Knife. We only use Tween and Timers but this library has a lot more functionality to offer. Checkout slide #15 for a gist of the modules that Knife provides.

Use Cases

The title changes colours. A timer is used to change the colour of the letters every 0.75 seconds.
Inside the game there is a countdown timer that goes from 60 to 0. User gains more time by making matches and the game ends when the timer touches 0.

Also remember that you have to remove or unset a timer. Timers are global objects, so if they are not needed in the new state or not needed after a certain point, unset them.

Check resources for the links on anonymous functions and first class objects.

tween0 “The Simple Way”

Tweens are used to change values over time. Say opacity from 100 to 0 in x seconds of time or say move an object from this position to that position in 5 seconds.

Colton then shows a screen where he moves a bird from one end to another in 2 seconds. He also points that it doesn’t do the task in exactly 2 seconds (took 2.01, 2.0089 etc). It depends on the specs of the computer. If it can maintain a high frame rate it’ll be more accurate.

Line 19 => The time the bird should take to move to the final position.
Line 31 => The final x position of the tween is set.
Lines 52 - 61 =>
On every frame just increment the position by the proportion of movement to the time elapsed. Just scaling to the ratio of elapsed time to move duration. math.min is used so that it doesn’t cross the end position.

function love.update(dt)
    if timer < MOVE_DURATION then
        timer = timer + dt

        -- math.min ensures we don't go past the end
        -- timer / MOVE_DURATION is a ratio that we effectively just multiply our
        -- X by each turn to make it seem as if we're moving right
        flappyX = math.min(endX, endX * (timer / MOVE_DURATION))
    end
end

You can assign multiple variables in a single line in Lua (similar to Python), just separate the values with a comma. A, B = 1, 2 for instance sets A to 1 and B to 2.

tween1 “A Better Way”

Lines 29 - 43 => Insert into a table the positions of 1000 bird with x as 0 and a random value for y (so that they scatter across the screen and cover the left hand side in it’s entirety). Also the rate at which it reaches the end is set. Notice that math.random is used twice. It’s used without any argument so that it returns a float. math.random(8) will return an int between 0 and 8, so by adding the additional float we are sure that rate is a float.

Lines 70 - 85 => love.update() same like before. Keeps updating time as long as it is below TIMER_MAX. Scale each bird’s x coordinate by time passed and the duration it should take to reach the end.

Stress Testing:- Enable FPS and go berserk. He tries rendering 10,000 birds and it works pretty fine. He increases the count and tries moving a million birds across the screen. The frame rate drops below 10.

You get a good idea of the limits of your game and the system as well.

tween2 “The Timer.tween Way”

Changing movement and opacity of the birds over time now. Doable using the previous approach but it gets dirty pretty quick.

Line 89 => You can set an opacity for fonts, graphics and images. Recall that you can specify an opacity when you define a colour to be used. Love uses the last colour set. So to modify the opacity of a drawable independent of it’s colour set colour to white i.e (255, 255, 255, desiredOpacity).

Code is very similar to the earlier example. In addition to the rate and x, we also store the starting opacity.

Lines 54 - 59 => We tween each bird in the birds table. Timer.tween() takes bird.rate and the values to tween. We pass in the bird currently being iterated upon and specify the final values as { x = endX, opacity = 255 }, so over the duration of bird.rate it consistently approaches the specified final values. We can pass in classes, tables and even variables to be changed inside Timer.tween().

With just 2-3 more lines of code, we can now tween two variables. This can be extended to tween even more variables with trivial changes.

Use Cases

Level text comes from the top to the center of the screen. It’s opacity and positions are tweened.
There’s a rectangle with stats. It’s opacity is tweened from 0 till it appears.

Code Run Through

StartState.lua Lines 24 - 31 => The colour table of the title text. Every 0.75 seconds the colours are cycled through. If a letter had colour1 it is now rendered with colour2, letters which were rendered with colour6 are now rendered with colour1 and so on. Check lines 44 - 52 for the implementation of the timer. The timer is removed on line 92 as the timer is no longer needed in the BeginGame state.
StartState.lua Lines 81 - 94 => If the user chooses to start the game, we tween our transition rectangle (just a rectangle with some text) from 0 to 255 (it means the rectangle was always present, it just wasn’t invisible until we increased it’s opacity). Also notice that we can pass in an anonymous function that is to be executed once Love’s done tweening the transition rectangle.

chain0 “The Simple (and Hard… and Ugly) Way”

Exploring ways to do stuff in succession. Say you want a non playing character to go walk 5 steps in 3 seconds, start a dialog with you, then do xyz. We want a way to reduce code complexity as well. If void of any special constructs, we’d model it using if statements. If the NPC has reached destination, open dialog box, if the dialog has been read, close it and so on. These conditions would be checked on every frame update. You can probably imagine the code getting complex pretty quickly.

Question:- Colton then shows a bird move around the perimeter of a screen. How would you do that?
You would have a bunch of if statements and variables to keep track of goals (top portion traveled along, right portion traveled along and so on).

Additionaly we need to account for time as well. The bird has to move across the screen in some predetermined time.

The code base becomes ugly pretty quickly. Also we now have to carry state (goals, positions etc).

In chain0 a very similar approach is used. Instead of an if else ladder, we use tables. Basically we store information that will help us do the above task in a table. We store the X and Y values that the bird should travel to in order.

Note that we could use a table because the 4 goals were very similar. Had they been different, we would have resorted to an if else ladder.

destinations = {
    [1] = {x = VIRTUAL_WIDTH - flappySprite:getWidth(), y = 0},
    [2] = {x = VIRTUAL_WIDTH - flappySprite:getWidth(), y = VIRTUAL_HEIGHT - flappySprite:getHeight()},
    [3] = {x = 0, y = VIRTUAL_HEIGHT - flappySprite:getHeight()},
    [4] = {x = 0, y = 0}
}

Chain.lua Lines 66 - 89

timer variable to measure time elapsed since the last destination (i.e reaching a point in the table). It is reset to 0 everytime a goal is achieved.
Iterate over the table and move the bird to the destination X and Y over time from the base (current X and Y).
Line 82 => Every destination coordinate becomes the base for the next goal.
Lines 75 & 76 => Update current X and Y coordinates of the bird in parts. Say the bird has to move 100 units in 2 seconds so at the 1st second it should be 50 units from the initial position. This scaling is handled by (destination.x - base.x) * (timer / MOVEMENT_TIME). (destination.x - base.x) gives the distance it has to cover and (timer / MOVEMENT_TIME) gives the ratio considering the time elapsed and the total time it should take.

chain1 “The Better Way”

We can use this approach for any operation including tweens.

Lines 30 - 47 => Use :finish on every tween to define what’s to be executed next. We further pass in tweens as anonymous functions on the completion of a tween.

Passing in a function to another function to be executed on the completion of the calling function (nested callbacks) can at times lead to callback hell (was a very common phenomenon in the JavaScript community).

There are ways to counter callback hell. The Chain construct can take in functions to be executed and will ensure they are executed in order. It’s very declarative and elegant.

The number of lines drop from 96 to 70. It’s also cleaner, promotes a declarative style to define async behaviour and is much more managable.

swap0 “Just a Board”

This update just renders a board and doesn’t actually deal with swapping tiles.

Generate sprite quads for the tiles from an image by evenly dividing the image into pieces like how we did in Breakout. Each block is 32 x 32 units.

3 or more of the same colour is a match. The lecture code doesn’t care about the inscriptions on the tile (cross, ovals etc).

generateBoard() returns a 8 * 8 2D array. It’s similar in spirit to the basic levelmaker of Breakout.

Why table[y][x] and not table[x][y]?
Look closely how the table we construct maps to the actual grid, you will notice that each row of the grid (therefore height) maps to the first index and so on of the our 2D table (table with tables).

It’s important to keep the fact that we index the table for the grid. So for the sake of convinience we do the bookkeeping in this way.

Lines 66 - 84 =>

Create a blank table for filling in the rows.
Insert into the this blank table an X coordinate, a Y coordinate and a random number that will be used to index into the quads to generate a random tile.
Table indices are 1 based but the coordinate system starts from 0. So we subtract one from the table index and then multiply by 32 (the width of a tile is 32). This gives the X and Y coordinate.

Lines 89 - 102 => drawBoard() takes in an X and Y offset and renders the above grid of tiles starting from that offset. It adds this offset to all the X and Y coordinates of the tiles and also uses the associated tile number in the quad table to render the appropriate sprite.

swap1 “The Static Swap”

How to swap two tiles?
Just swap two tiles (like how you interchange the values of two ordinary variables). In the tiles table we just need to swap places. We use a third variable (that acts as a holder) to swap the values. If we don’t use this temporary variable or holder we’ll lose information pertaining to one of the tiles.

In Lua you don’t have to use a temporary variable to swap (recall that you can assign multiple variables values. Eg x, y, z = y, z, x would swap x’s value with y, y’s value with z and z’s value with x).

A rectangle indicates which the tile the user is on. Arrow keys are used to move the marker across tiles. Currently there are no constraints, any tile can be swapped with any other tile.

If you were to implement constraints (only adjacent tiles can be swapped) you’d probably use the code below. If a tile is adjacent (top, down, left or right) it moves by a single step. So the sum of absolute differences between their X’s and Y’s would be 1. If we were to sum the difference of diagonal tiles we would get 2. Think about it and enumerate a few cases on a piece of paper.

-- To check if two tiles are adjacent (top, down, left or right) or not
if math.abs(tile1.x - tile2.x) + math.abs(tile1.y - tile2.y) == 1 then
 -- swap tiles
end

We now have to check if a tile is highlighted or not (user presses enter on a tile), we store this boolean flag in a variable. If a tile is highlighted we render a semi transparent rectangle over a tile. The rectangle is rounded for aesthetics.

The first time the user presses enter, it highlights a tile with a semi transparent rectangle and he gets another red marker which he can move around to choose another tile. After he places the second marker on another tile and presses enter again, our game swaps the tiles.

Lines 189 - 200 =>

We set an opaque red as the colour. We also set a lineWidth of 4. So when shapes are drawn they are thicker than usual.
On line 197 we draw a rectangle like usual but we pass in line so that the rectangle is hollow i.e the rectangle just has a border. By using line it uses the lineWidth we set earlier. Additionally we also pass in 4 as the last paramter so that the rectangle is rounded.
Always reset the colour after your change it. If you don’t you will have to be extra careful to ensure that it doesn’t render everything in the last set colour. You could also set the colour everytime just before you use it. That works well too.

Lines 58 - 75 => Takes care of input handling. It changes the appropriate tiles on user inputs.

Lines 85 - 101 => This block is entered when the user has highlighted a tile and then presses enter with the marking box on another tile. It just swaps the tiles.

Swap the tiles using temporary holders. We also need additional data like gridX and gridY. We’re using temporary holders for them as well. He also mentioned that we swap stuff in the board (lines 94 & 95) before swapping the coordinates and the positions (lines 98 - 101) as it’d give birth to some weird bugs.
We also set highlighted to false. The red rectangle (marker) now is set to tile2 (the tile selected by the second marker).
It is worth knowing that we could have used simple multiplications (by 32) and additions (the offset) as well to derive stuff from the table indices instead of swapping internal data like coordinates.

swap2 “The Tween Swap”

This update enables tweening the tiles to their new positions instead of instantly swapping them.

It’s a relatively easy to achieve effect. Just tween tile.x to tile2.x, tile.y to tile2.y and tile2.x to tile.x, tile2.y to tile.y. Tween the positions of the tiles to be swapped to the position of the other.

Lines 99 - 102 => Implements the logic described above spread across a duration of 0.2 seconds.

Calculating Matches

Question:- Any ideas on calculating matches?

A student suggested that we could look at adjacent tiles and if they are of the same colour, find the direction and then go recursively deeper in that direction. Colton pointed out that it’s trickier to implement and it’s probably inefficient as well.

Go over each row horizontally first. If consecutive tiles are of the same colour keep incrementing matchNum until the pattern is broken. If matchNum is 3 or more it’s a match. Also if you’re reached the end of the row stop and skip to the next row. You also have to reset matchNum as you’ll be starting across a fresh row.

Do the same thing vertically across the columns of the table.

If a match if found (3 or more) append them to the table containing the list of matches.

There’s also a possible optimisition. If you’re at the 7th tile and the match count is 0 or 1, you don’t have to check the 8th tile as it wouldn’t form a match anyways.

See slide #25 to slide #66 for a visual explanation. There’s hardly any text you can quickly view it as an animation.

Student Question => If we delete the tiles as and when see matches and there was an intersection of matches (forming a T shape or a cross)?

Colton’ Reply => If we cleared matches as and when we encountered them, we wouldn’t see the second match as one or more of the common tiles of the matches would be cleared and it would no longer be a match.

The current algorithm we implement goes over the entire grid before deleting matches so it would not miss it. It would see the first match of the intersection while going horizontally and then the second one while iterating vertically over the grid.

Colton also suggested that we could reward the user with more points in case of an intersection, clear the entire row and do those sorts of things.

Board.lua Board:calculateMatches()

Variables for current colour, matchNum (how many consecutive of the same colour seen), matches etc.
A branch for shortcircuiting at 7th or greater tile.
Loop to keep counting the same colour if found.
If a match is found, insert it into the matches table. Insert into the table all the tiles that are part of the match. We insert all the tiles because it’s easier to find if a tile is in the match table or not. If we inserted match ranges (index positions), we would have to write logic to check if a given tile is a part of the range or not. This takes a little more memory but simplifies stuff quite a bit.
We iterate from 2 to 8, so if the last tile of the horizontal row is a part of a match, we will not reach the else block that takes care of calculating the matches since we would have exhausted the for loop. So there’s an if block on line 95 that handles this corner case and adds the match to the matches table.
Exact same logic for the vertical matches. Just x & y are inverted.
self.matches = matches so that we have a reference to all the matches. They’ll come in handy later on when we want to remove tiles.
In the end this function either returns false (if there are no matches) or the matches table. This is a neat API design as we can directly use if Board:calculateMatches().

Removing Matches

You should have realised by now that all the matches are part of a 2D table (a table of matches and each match has a list of tiles). We first have to set all tiles in the matches table to nil. It has the effect of removing the tile from the grid.

Once the matches are removed, we have to bring the impact of gravity. Higher level tiles that still exist should fill lower level spaces.

Jist of the algorithm => Start from the bottom of each column and find an empty tile (i.e look vertically and start from the bottom). If an empty tile is found go upward and look for the nearest tile and swap the tile with the empty space. Then start with the position just above the space you encountered and not the location of the tile you swapped it with because we could still see spaces between them. We don’t have a fixed number of items to check (we just have to ensure that the tiles are at the bottom and if spaces exist, all of them should be above them). So we resort to using a while loop rather than a for loop. It now has a similar look and behaviour of the bubble sort algorithm. If no more tiles are found since the highest tile in the column, the column needs no more swapping.

A student made a nice suggestion that we could just shift all tiled cells to the bottom without having to go through all this. It’s a neat idea. It’s similar to getting rid of spaces in strings and as a result everything moves to the left anyways.

I think Colton took us through the approach described above so that we could get a clearer picture of what’s happening.

Board.lua Lines 182 - 209

Line 183 A flag for space (empty tile) or not.
Lines 191 - 207 If a space was encountered the flag is set to true. If a tile’s encountered it means that there’s one or more spaces followed by a tile. We then enter the if branch under line 193, tween the tile to fall and then reset space = false. So we’ve accounted for the lowest space and the tile vertically just after it.
Notice that we use an outer for loop from 1, 8 (for the columns) since the number of columns in known in advance. The inner loop is a while loop (for going over each column vertically) as the number of tiles we have to fix is not known in advance. Each columns might need different number of swaps, some might not even need any.

See slide #67 to slide #93 for a visual explanation. There’s hardly any text, you can quickly view it as an animation.

Check resources to see bubble sort in action. Also check out string strips in Python to better understand the idea suggested by the student.

Replacing Tiles

Check from the top if there are empty spots. Remember that gridX and gridY are coordinate values, x and y are just indices in the grid table. Set the empty spot to a tile using x and y so that the internal data structure is correct. This is essential as the tile has to be stored in the correct place in memory.

Once that’s done we can manipulate it’s gridX and gridY values to give the right effect, render it in the right place, create visual effects like spins etc. We currently tween it’s gridY value to make it look like it’s falling into place from the top.

Also the inputs are disabled while replacing tiles to avoid bugs.

PlayState.lua Lines 189 - 192 => 50 points for each tile in a match.

PlayState.lua Lines 198 - 211 => We first get the tiles with their respective positions and target positions using getFallingTiles() and tween them to their new positions over 0.25 seconds. After that’s done we also need to get new tiles and tween them to their respective positions (tween gridY to make it look like it’s falling). After that’s done we recalculate matches just in case the new tiles create new matches. calculateMatches() recursively keeps calling itself everytime it finds a match on the new tiles. Say with the new tiles spawned it found a match, it destroys them and generates a new set of tiles again upon which it has to check for matches again.

Palette

Taking some sort of image and using some arbitrary number of colours from it. You basically use a limited number of colours that are quite likely to go well with each other.

He briefly mentions about a famous colour palette - DawnBringer’s 32 Colour Palette.

Dithering => Draw two coloured pixel by pixel interleaved (alternation colours colour1, colour2, colour1 and so on). The end result looks like a new colour.

Check slide #109 for the resultant colour by dithering each of the 32 colours with each other. The first row and column are the individual colours. The intersections are the result of the row and column.

You can create loads of cool stuff by using a limited colour set and dithering. Check out slide #110, all of those scenes were created by dithering a small palette of 16 colours.

There’s a famous 32 colour palette by Dawn Bringer which can be put to good use. You can convert images to a particular palette as well. Colton then shows how an image of a cat (with 1000’s or millions) would look if converted to this colour palette. It looks very similar to a cat but had a very inconsistent and blotchy look (anything with blurs develop patches). He then showed an example where he converted a comic with many flat colours into this colour palette. It had a nice feel to it. The gist being that a particular colour palette will not work with all images.

This game and the previous game (Breakout) use this very same 32 colour palette.

By imposing the constraint of a limited colour palette you give the game a consistent look. Can also be used to give a nice retro feel. Old games were limited by hardware. They could only display a limited set of colours and hence resorted to colour palette and dithering.

Palette Swap
Create a gray scale version of a sprite. Have distinct parts which you can identify. Use different colours for each of those distinct parts. The same sprite could just be used as a different game object. Mario actually did this. They had the same sprite but used a different colour palette to colour them.

I’ll just quote Wikipedia here.

A palette swap is a practice used in video games, whereby a graphic that is already used for one element is given a different palette, so it can be reused as other elements. The different palette gives the new graphic another set of colors, which makes it recognizably distinct from the original. Palette swaps are commonly used to distinguish between first and second players, for creating visual hierarchies, and for making visually distinct areas for levels in games.

Assignment

He briefly explains about the assignment.

If you want to implement mouse functionality, remember that you will have to use the Push library to convert the actual coordinates on the screen to a set of coordinates that are scaled to the VIRTUAL_WIDTH and VIRTUAL_HEIGHT.

End

GD50 Lecture 03 - Breakout

Thu, 22 Feb 2018 00:00:00 GMT

Resources

Slices in Python.
Why Use Sprite Sheets? - Link 1, Link 2.
Intro and Exercises in Particle Systems - This is awesome.
ASCII Explained
Folders Under AppData in Windows
./local/share/ in Linux
Collision Detection for Fast Objects.

Breakout State Flow

Quick overview of possible movements between states. It’s quite clear from slide#7 that it is possible to go to the HighScoreState from the StartState and vice versa but it’s not possible to directly enter the PaddleSelectState from the HighScoreState. He also briefly explains what happens in every state. For instance if the ball goes below the paddle and the player loses a life, the game transitions from the PlayState to the ServeState. I don’t want to go into these details for the sake of brevity. Essentially you draw a detailed state diagram to get a better picture of the flow. It’ll also help you refactor your code better. To know more use cases of a state diagram see Section Pong-9 of Lecture Pong

breakout0 “The Day-0 Update”

Organise your project into folders. See slide#9 and you’ll get a clear idea.
Global tables are initialised for fonts, textures and music. Within the music table, you notice that the background music is separated from the other audio files by an extra line break (line 85 of Main.lua). He mentions that using a ResourceManager class is a better way ahead.

A screen is setup like the usual lectures. Instead of having the constants in the same file (VIRTUAL_WIDTH and VIRTUAL_HEIGHT) all the constants are moved to src/constants.lua.

A state machine is setup and StartState is entered into. Like last lecture a global input table is setup (for keypresses). A function displayFPS() is written so that the frame rate is visible on the screen instead of having to see the terminal. It’s a common practice to display FPS on the screen during development. You can read more about it in my notes on Pong.

Instead of having to require multiple files repeatedly a src/Dependency.lua is used. It imports all dependencies internally and by importing it, you implicitly import other dependencies. All states are moved to src/states to make it modular. It might seem like a lot of folders at first but it simplifies and keeps things organised especially in larger games.

On line 181, you will notice the background image being rendered. The virtual heights and widths are being divided by the height and width of the background image to get the scaling factor. Also notice that 1 is subtracted from the background image’s height and width. This creates a larger than necessary scaling factor. Ensures that the image is stretched and always fills the screen.

breakout1 “The Quad Update”

local highlighted in state/StartState.lua is used to keep track of the highlighted menu option (start or highscore). It is set to 1 to highlight Start by default.

Lines 23 - 28 => the highlight variable is toggled between 1 and 2 every time the user presses the up or the down key. In this case we only have two menu options and can simply toggle. In other cases we’d have to increment and decrement highlight by 1 on up and down press respectively.

Escape key presses aren’t global anymore. Pressing the escape button in some states switches to the previous state (acts like the back button).

The highlighted option is given a different colour. Remember that Love2D is kind of like a state machine and uses the last set colour for rendering text, so we explicitly set the colour every time we want to render something. Also see line 53, the last of the four 255 sets the alpha of the colour. Setting it to 255 makes it as opaque as possible. The value can be tweaked to give your text tints.

Sprite Sheets

Instead of having multiple image files for each of your asset, put them all in one giant image. We can then load the single file into memory and get parts of the drawable (image) and render them. Basically select a sub portion of the image and ask Love to render it at your desired location. See slide#11 and 12. Sprite atlas is a synonym for sprite sheet.

Check slide#13 for the function definitions.

There are tools to generate quads (parts of sprites which contain images). They prove to be useful when you have image files with many sprites especially when the dimensions of the sprites are uneven. They can help you generate dimensions to get the respective sprites.

Every file from this point onwards will be required by src/Dependencies.lua.

Util.lua => has a function to generate quads. Basically iterate over the atlas and divide them into tiles of equal width and height. The code is pretty straightforward. It stores the quads in a table and returns the table.

On line 42 => Colton has defined a helper function that is similar to slicing in Python. Lua doesn’t provide such an option by default. On line 45 is the meat of the function, if starting point isn’t mentioned, it starts from the beginning, if the end isn’t mentioned it goes on till the end (represented by #tbl. # gives the size of the table). The third argument of a for loop in Lua increments the variable by that much (if you’ve used loop in other languages, you probably have used i++ or i = i + 2). It’s just that. So in short it’s a typical for loop in a language like C, JavaScript or C++.

Line 57 => defines a generateQuadPaddles to get the paddles. Notice the image file and see how they are spread across. Each block is 16 px tall so we start y at 64 and x at 0 so that we get the first paddle on the 4th row. Adjustments are made to get the paddles on the left as well. There are 4 such rows, instead of having to repeat the logic, a loop is used, y is incremented by 32 to index new rows and x is set to 0 to get the first paddle. They are stored in a table as usual and returned.

Line 65 => of Main.lua you can see the function (also pass in the main sprite sheet to the function) in use. A global table gFrames is defined. We store all the quads that pertain to paddles in this global table.

The Paddle class has variables skin and size that keep track of the colour and size. If you notice, we load the paddle quads in the order in which they appear in the sprite sheet. There are 4 sprites of each colour and there are 4 colours. The second paddle of the 3rd colour can be accessed from the table by indexing #10 from the table (2 colours skipped and 1 skip from the 3rd colour so ((2 x 4) + 2)). The other bits should make you feel right at home. Very similar to the previous lectures (getting input, updating x coordinate, containing movement within the boundaries and render (this time using quads instead of the entire texture)).

PlayState.lua line 20 => A new paddle is initialised. It also has logic to play and pause the game.

Also by generating quads there’s one less headache now. No need to name, sort and organise images. We can simply load then in one table and them random access them from the table using indices.

Check resources to know about Python slices and the advantages of using sprite sheets.

breakout2 “The Bounce Update”

In Util.lua a new function GenerateQuadBalls is defined. It generates the quads pertaining to the balls from the atlas. It’s very similar in fashion to how quads are generated for the paddles. Notice in the sprite sheet (slide#12) that each ball is around 8px in height.

A global table is used for all quads. Colton says it’s more organised and neater.

Ball objects are initialised with a skin. It also implements a generic AABB collision. It can take a target (any drawable) and check if the ball collides with it or not. An instance the Ball class can get easily get the required quad by gFrames['Balls'][self.skin]. A random number is passed while initialising the ball so that it gets a random colour.

Ball.lua line 68 => Walls start deflecting the ball now. If the ball hits the top, we let the X velocity be and just reverse Y velocity. If we hit the left or right edge, the Y velocity remains, just reverse X velocity. Recall in Pong we actually had to reset the position during a collision else we’d risk getting stuck in an infinite loop. The case where the ball hits the paddle at a weird angle is yet to be fixed.

PlayState.lua => Initialises a ball, checks for collisions with the paddle.

breakout3 “The Brick Update”

generateQuadBricks() of Util.lua is used in the gFrames table. It internally uses the GenerateQuads function to split the atlas into dimensions of given width and height. We pass in the width and height of the brick. You probably figured out that the function would return many more quads than required (that too wrong ones) as we use a crude approach to split the sprite sheet using the dimensions of the brick. The quads it returns are no doubt uneven and pertain to the entire sprite sheet but we don’t care. We’ll slice it to get the first 21 (the bricks) and the rest will be discarded. This is a one time thing that happens during the start of the game and it really doesn’t matter even if it’s a little inefficient.

Bricks have a member variable called inPlay. It’s a flag that identifies if it’s alive or not. Using this flag is a simple workaround for having to deallocate the brick (if it’s hit, it probably needs to be removed from the table that holds all bricks during gameplay). This flag is used with an if condition before updating and rendering (we selectively render and update bricks that are still inPlay). For a small game this approach works. For large games we’d have to deallocate memory. On brick hits, we just have to set inPlay to false.

Line 49 => We index the appropriate colour and tier (bricks of varying health or strength levels, differentiated by colours) like how we did for paddles to get the right quad from the quad table (offsetting by multiplication and addition).

PlayState.lua line 34 => LevelMaker.createMap() a function to create a table of bricks to be used anywhere. Additionally we also iterate over all the bricks and check if the individual brick collides with the ball. We also render bricks selectively using the flag described above.

LevelMaker.lua line 26 - 47 => All it does now is randomly choose the number of rows and columns and places the bricks along the correct coordinates to center align all the bricks. It’s pretty simply as all the bricks are of the same colour and it fills the entire grid with bricks. Lines 34 to 43 are the most interesting lines in this file as of now, it initialises Bricks after taking top and left padding into consideration. The lines are well commented and are easy to understand. Fancy patterns and colours will be covered in eventual updates.

breakout4 “The Collision Update”

Currently the ball just passes through a brick. It should deflect after hitting a brick. In addition we also need to know where the ball is coming from, so that we can get it to bounce in the right direction. (If the ball hits a brick from the left, we just deflect it towards the right etc). We need to find out which portion of the ball isn’t inside the brick and we’ll also be using dx to know the direction it was travelling in before hitting the brick.

When the ball comes in contact with the corner of a paddle, it should take a sharp turn or cut. However if it hits an area near the center, it shouldn’t deflect by much of an angle. See slide #17. A good measure for this intensity is the distance from the center of the paddle.

PlayState.lua line 64 - 71 => There are two if else branches which we’ll go over one by one.

self.paddle.dx < 0 dx not being zero indicates that the paddle is in motion. It being less than 0 indicates that it is moving towards the left. self.ball.x < self.paddle.x + (self.paddle.width/2) is used to check if the ball’s x coordinate is less than the x coordinate of the center of the paddle. The second line sets a dx of 50 in the left along with 8 times the distance of the ball from the center. Distance from the center is given by (self.paddle.x + self.paddle.width /2 - self.ball.x). The 8 times creates the amplification (sharper angle and faster speed) effect.
The next else if branch is very similar. He’s just made changes to directions (dx should be positive for instance) and used math.abs to convert the negative difference between the center and the ball to a positive dx.

For deflecting the ball off the brick in the right direction we use a simple rule based system. It’s simple but works most of the times (only fails if it hits the brick at weird angles). For the sake of time and brevity Colton decided to use this approach. See slide #18 for a picture and some more details. He pointed to another tutorial that is quite complex but solves this problem accurately. Relevant links to those tutorials - this and this.

PlayState.lua Line 80 => checks if a Brick collides with a Ball. Lines 95 - 121 take care of deflecting the ball in the right direction. There are 4 branches in the if else ladder. Let’s go over them one by one.

Line 95 => checks if the ball’s x coordinate it less than the brick (meaning the ball is approaching the brick from the left). Notice self.ball.x + 2 < brick.x, 2 is added to the ball’s x coordinate to account for edge collisions (top left or bottom left corner of the brick). If the ball hits say the bottom left corner it gives more priority to the coming from right. The jist lies in self.ball.x < brick.x, the + 2 is a small fix.
98 & 99 => If the above condition is true, just reverse the velocity of the ball and shift it outside to avoid pesky infinite loops (recall these infinite loops were covered in Lecture 01 - Pong. We subtract 8 because that’s the width of the ball and moves it away from the brick. We could even use self.ball.width instead.
Line 102 => self.ball.x + 6 (same offset of 2 but on the right, should have been self.ball.x + 8 but we reduce 2 being the fix for edges and it becomes + 6). We shift by 32 as the brick is 32 units broad.
Similar treatment for top and bottom collisions. Reverse dy instead of dx and move by 8 and 16 units (the brick is 16 units tall).
Line 124 => Increase the speed of the ball with each hit.

These collision detection rules will fail once dx or dy becomes too high (the ball moves a lot each frame, one frame it might be outside a brick and in the next frame it could have traversed multiple bricks and get past them. Recall that we test for detection every frame, since the ball isn’t in contact with a brick, we won’t be able to detect the collision). It just skips over them.

Colton explains a better way to detect collisions at the cost of more resources. Jist of the better idea is as follows.

Detect collisions along the entire path travelled by the ball between the last frame and current frame. You have dx and dy so go backward to the last frame.
From there on make small additions to the ball’s x and y values until you touch dx and dy (i.e reach the position in the latest frame after dx and dy are added). Keep checking for collisions after these small additions that is.
This way you’ll ensure that you don’t skip through objects.

Check resources for the link on detecting collisions when an object is too fast.

breakout5 “The Hearts Update”

StartState.lua 35 - 40 => Switch to the serve state if option 1 is selected from the start state. Pass in health, the paddle, the score and bricks.

Only the assets are global in this game. Every state is passed variables that it needs. Recall our state machine can take in a table. Even web frameworks like React work in a similar fashion. It passes the required data to the state instead of maintaining globals. Keep in mind that if a certain state requires some data, the present state which transitions into that certain state will also need to have access to that data. So you might have to pass data via various states so that data can trickle down to that certain state.

The serve and play state probably need bricks, a paddle, hearts for indicating life etc but the high score state needs only the highscore. By passing in what’s required rather than using globals you keep stuff clean and encapsulated. You also get a bird eye view of what each state needs.

ServeState.lua 39 => On state transition we additionally pass in the health as well.

Main.lua

Has quads for hearts. It was generated by splitting an image evenly (recall how we split the bricks).
208 => Function to render the hearts. It first renders the hearts the user has left and then renders the hearts he’s lost. To render the hollow hearts (missing health) the difference between total lives and lives held is taken, so if the user hasn’t lost any life the for loop fails in it’s very first iteration.

PlayState.lua

81 => Every brick hit simply add 10 to the score. Will be improved in the future updates.
132 => Check if the ball goes below the screen, reduce health then switch to serve state. If lost all lives switch to the game over state. Pass in the variables that the respective states require.
164 => renderScore is used to display the score (very simple and the definition can be found on line 239 in Main.lua).

Any variable defined without using the keyword local becomes a global and is accessible from anywhere in the program. So it’s a good practice to use local to avoid bugs that arise from unintentionally changing global variables.

breakout6 “The Pretty Colours Update”

Flags for overall map shape and individual row properties. These flags are used to create random levels that look handcrafted (The code doesn’t play with the overall map shape at all, you can safely ignore NONE and the 2 PYRAMID flags).

LevelMaker.lua

Line 40 => Ensure that the number of columns are odd as even columns can lead to asymmetry.
Line 45 => Decide the tier based on the level currently in subject to a maximum tier of 3 (we just have 3 tiers).
Line 48 => Colour cycles are set and restricted to 5 and decided based on level as well. Lines 45 and 48 have the effect of increasing a tier and resetting the highestColor to blue every 6 levels.
The flags skip and alternate are used to create fancy patterns. If skipFlag is true (set randomly on line 65), every other slot of the row is empty. If alternate is true (set on line 68), the colours of the brick alternate and if solid is true, the entire row is of the same colour. Additionally the number or rows are also random.

Just simple concepts like skipping and alternating colours are used for our procedurally generated game maps.

Student Question:- What happens if the ball is between a brick in a particular frame?
Like mentioned earlier our code fails when delta is too high. So if it’s between a brick, it’ll trigger AABB collision. Recall that we try finding out the portion of the ball that’s outside the brick. No portion of the ball falls outside the brick hence the last default branch (bottom collision) is triggered. It behaves as if the ball has been hit from the below.

breakout7 “The Tier Update”

Now the Brick class makes use of colour and tier while updating and rendering a brick. Checkout the new updated Brick class.

Brick.lua line 44 - 58 => Get a brick of a higher tier (strength level) to a weaker state represented by a different colour. If it’s in the weakest state and gets hit again, set inPlay to false so that the brick doesn’t get rendered.

PlayState.lua line 81 => Simple arithmetic based on colour to update the score once a brick is destroyed. So different colours and different tiers get different scores. The base tier is zero so you don’t get the additional 200 points bonus on hitting it. You can play around and try out different rules to reach an optimum scoring system.

breakout8 “The Particle Update”

Particle systems make a variety of effect achievable. These effects are normally difficult to achieve using sprite animations. Things like an organic fire, flowy or smooth effects are normally achieved using the particle system.

How would you create a fire? Lots of fire particles in an area (representing the center) and gradually reduce the number as you move further away. They probably float upwards and then disappear. In other words write some logic to spawn fewer and fewer particles from a given point, set some -ve dy so that they float upwards and then fade away. You could additionally use a timer to transition the colour from glowing red to yellow. As some particles move up they could turn brown or black (smoke) and disappear from the screen.

We use Love’s integrated particle system. It takes in a texture as a foundation and the number of particles to emit. The number of particles the particle system can emit, their speed etc ultimately decide the possible effects you can create. Particle systems have loads of functions and you can check the ones provided by Love here.

Every brick when hit, needs a particle system of it’s own of their own colour (red brick hits should give out red particles).

Logic for the particle system used in our game:- Randomly generate particles close to our brick, fade away after a certain amount of time. They are of the same colour as the brick in the start and become transparent before they ultimately fade away.

Brick.lua line 20 - 51 => Defines a colour palette in the form of RGB triplets for each brick.

Colton mentions the importance of having a limited colour palette. It gives a cohesive look and even gives a retro like touch. Makes it easier to choose colours as it limits the choices you have. Can be a huge time saver.


  -- particle system belonging to the brick, emitted on hit
    self.psystem = love.graphics.newParticleSystem(gTextures['particle'], 64)

    -- various behavior-determining functions for the particle system
    -- https://love2d.org/wiki/ParticleSystem

    -- lasts between 0.5-1 seconds seconds
    self.psystem:setParticleLifetime(0.5, 1)

    -- give it an acceleration of anywhere between X1,Y1 and X2,Y2 (0, 0) and (80, 80) here
    -- gives generally downward
    self.psystem:setLinearAcceleration(-15, 0, 15, 80)

    -- spread of particles; normal looks more natural than uniform, which is clumpy; numbers
    -- are amount of standard deviation away in X and Y axis
    self.psystem:setAreaSpread('normal', 10, 10)

The code snippet given above are lines 66 - 81 of Brick.lua. The life time, area spread and texture of the particle system is set. It’s a part of the init function because each brick has it’s own particle system.

Lines 92 - 102 => set the colours for the particle system. 2 colours are set. The transparency (alpha) varies between them. 55 * (self.tier + 1) makes a higher tier more opaque hence brighter. All this code does is specify two colours (using the colour table defined in lines # 20 - 51) with different levels of transparency. The particle system creates particles of the first colour and transitions them to the second colour (transparent in our case) during the lifespan of the particle. 64 particles are emitted.

Line 152 => clearly shows that the particle system is initialised with the center of the brick as it’s base (recall that the dimensions of the Brick are 32 and 16).

Check resources for a detailed introduction and a few exercises on Particle Systems.

breakout 9 “The Progression Update”

This update starts showing levels to the user. All we have to do is store a number. The number is incremented when all the bricks are destroyed (hence all the bricks have inPlay set to false). A new state victory is created. The game transitions into the victory state when the condition above is fulfilled.

StartState.lua 40 => When changing to a new state just pass in the level in the table. Any time we change state, we just pass in an additional level variable.

PlayState.lua 204 => A function to check victory. If a brick is in play return false else return true.

PlayState.lua 88 => If victory is true, play a sound and transition to victory state. All the victory state does is display a message. Pretty straight forward. Also gives an option to resume game by proceeding to the next level.

VictoryState.lua 35 => Notice when we transition from the victory state to the new state again, the victory state increments the level and also makes use of the level maker to create a new set of bricks.

breakout10 “The High Score Update”

Check out the file system related functions in slide #27.

Most operating systems have specified directories to store application data. Love only allows writing files to the directory assigned by the operating system for such purposes. You can create subdirectories and files within this directory.

For instance on Windows the directory is C:\Users\user\AppData\Roaming\LOVE. The directories for other operating systems can be examined here. This directory is already hard coded by Lua. So you will only need to use love.filesystem.setIdentity only when you want to write to a subdirectory within this directory (you mostly won’t need to).

We’ll just have a table (table of tables holding name and score) and sort them by score. While writing to a file we’ll need to convert it to a string. Colton decided to take a simple approach while storing this info in the file. So 10 high scores would result in 20 lines in the file.

name
score
name
score
.
.

main.lua line 213 => Create a folder named breakout in the local storage directory. If a file named breakout.lst doesn’t exist, create some dummy high scores and save it to that file (also called seeding). If the file already exists iterate over every line using abstractions provided by Love. If it’s a name just take the first three characters (we only accept 3 characters and this serves as a safety check against long names). If it’s a number read the line (you will get a string type) and convert it to an integer. If this sounds too simple, you could write some code for error checking as well. You could check if the file has been tampered with or if a score is a garbage value, handle it elegantly and prevent it from being rendered and so on to make it more robust.

Check resources and read about these special folders and their purposes.

breakout11 “The Entry Update”

Allows the user to enter his name when he gets a high score. An arcade game like style is used (only 3 characters for storing a name, use up arrows to traverse from A to Z and left and right arrows to change position).

How should we move from A to B, back to A and so on in response to up and down key presses?
A student suggested that we could have a table with A-Z and index the next or previous element. This is bulky and Colton proceeded to describe a better way which I’ll summarise below.

A new state EnterHighScoreState is created. Check out slide #28 to see how it looks. It’ll also make following along easier.

A table local chars is used to store the ASCII values of the three characters. local highlightedChar is an integer that keeps track of the desired character (highlighted on the screen) amongst the three characters. We index the chars table using highlightedChar and change the value accordingly. If the left or right arrow key is pressed, it decrements and increments highlightedChar and if the up or down arrow keys are pressed it changes the value of chars[highlightedChar] by +1 or -1 effectively going to the next or previous alphabet.

If we exceed 90 (z in ASCII) we must wrap around to 65 (a) and if we try going below 65, we should display z. These edge cases are taken care of using if conditions (on line 74 and 79 of EnterHighScoreState.lua).

To reach this state the user should make it to the high score list. So it needs to have the current high score list and also the current score. We have to pass these along the possible states.

We start by comparing the current high score from the bottom of the list and place it below the score higher than it and shift everything below it one spot down.

Check resources to know a little about ASCII.

breakout12 “The Paddle Select Update”

Just displays two arrows for directions and renders a paddle. Every time the right or the left key is pressed it increments or decrements a counter and indexes into the quad table to choose the appropriate paddle. Also the direction hint arrows change (between transparent and opaque) to indicate if the user has reached an extreme and cannot scroll further for more possible paddles in that direction. Check slide#29 to get a better picture of what I’m saying.

Once a colour is chosen by the user, initialise the paddle with the respective colour (recall the choice is stored in the counter) and the paddle can be passed to the various states you might transition into.

breakout13 “The Music Update”

Sound effects for the game.

He also mentioned to have a look at the code that recovers the life of a user by a heart once a certain threshold is reached. This rewards the user for his efforts.

End

GD50 Lecture 02 - Flappy Bird

Mon, 05 Feb 2018 00:00:00 GMT

Resources

Examples of a Few Parallax Sites - Have a look if you want to get a feel of Parallax. Skip if you already know what Parallax is.
Iterating a Table in Lua.
Basic Sound Wave Types - With animations and sound demos.

Optional Reading

These links were suggested by Colton during the lecture. They are purely optional.

How to Make an RPG
He uses Lua with his own custom engine (similar to Love) to make a Role Playing Game. Colton mentioned that he got his hands dirty with Lua using it.
Game Programming Patterns
Great general guide to learn game development concepts for large scale game development. Personal Note:- I’ve read a few of his blog posts and I can vouch that (Bob/Robert) Nystrom is awesome.

Please note that this lecture builds on a lot of foundation that was set in the previous lecture.

bird0 “The Day-0 Update”

love.graphics.newImage(path) - Loads an image from a file that we can later use to draw to screen.

He sets 512px and 288px as the virtual dimensions. He chose them as it worked well with the assets and they also have a 16:9 resolution. You are free to choose whatever you want after a few trials.

A filter was set (The filter decides what’s to be done when you upscale or downscale a texture). You can read more about them in my previous blog post.

Covers quite a few things that are very similar to the first lecture.

-- images we load into memory from files to later draw onto the screen
local background = love.graphics.newImage('background.png')
local ground = love.graphics.newImage('ground.png')

The keyword local binds the scope of the variable to the current file. background and ground cannot be accessed outside the file and it also avoid variable collisions. If another file happens to use the same name, we won’t run into the risk of changing the global variable with the same name.

love.graphics.draw() can be used to draw anything that is of type drawable. Images come under the type drawable. So keep in mind that background and ground can now be drawn to screen.

He recommended using aseprite to create sprites. Other popular options are Photoshop and Gimp.

bird1 “The Parallax Update”

Parallax can help giving an illusion of movement. Different frames of references move at different rates. For example you are travelling in a car. The mountains in the distant move rather slowly when compared to the rate at which the fences on the road move. You get a better feel of movement and distance as well due to this.

Two variables backgroundScroll and groundScroll are declared to keep track of how much the respective images have moved. The background mountain image and the ground image slowly move across the screen and wrap around the screen when a certain point is reached. These variables will be used to compare against those certain points.

-- speed at which we should scroll our images, scaled by dt
local BACKGROUND_SCROLL_SPEED = 30
local GROUND_SCROLL_SPEED = 60

Two variables to store the rate at which the background and the ground move. It’s a convention to set variable names to all caps if they are not going to manipulated i.e they are some kind of constants.

Now he shows the technique he uses to get the mountain and the ground to infinitely scroll across the screen. The actual image has two copies of the same mountain next to each other. So it creates the effect of wrapping around the screen. When a certain point is reached we just snap it back to the original position. Have a look at the picture below.

It makes logical sense to snap back the picture once the first mountain reaches the end of the screen (if we continue going forward and if the left corner of the screen touches the second mountain the image will no longer cover the entire screen).

However we needn’t be so pedantic while setting the looping point of the ground’s image. If you notice it has a pattern that keeps repeating itself anyway. We shouldn’t notice any difference even if we set an arbitrary value.

backgroundScroll = (backgroundScroll +
                   BACKGROUND_SCROLL_SPEED * dt)
                   % BACKGROUND_LOOPING_POINT

The code above shows how it’s achieved. We curb the image’s movement beyond a certain point by using % BACKGROUND_LOOPING_POINT on the probable future position of the image.

There are ways to save memory (using an image that has twice the width of the regular image doesn’t seem very efficient, does it?). You could use a regular image and render it twice to create the desired effect. Then once a certain point is reached snap them back to their original position. Since there’s only one image and both the drawables point to the image, memory requirements are lesser. This same technique can be applied even for rendering more copies of the same image (it’ll be easier to appreciate the savings in memory from texture sizes now. You can get the same effect without using an image that is 4 times as wide as the normal image).

He shows a YouTube video where they’ve hacked the camera of a few games. They can now see things outside the reach of a normal user. By seeing things beyond the normal boundaries they spot a lot of tricks used in game development to keep memory requirements low. For example:-

A game only had the face of mountain rendered. It looked gigantic but on rotating the camera one could see that it’s actually hallow. By not having to render those points and vertices a huge deal of memory is saved.
A monster is positioned in a shop in such a way that you can only see it’s face. On digging deeper they notice it doesn’t have legs but the user is oblivious to that fact.

In short games give the illusion of doing lots of grand things by using neat tricks.

Check resources for a few parallax sites.

bird2 “The Bird Update”

He creates a bird class. Recall from the previous lecture that it’s a convention to start the name of class with an upper case letter.

function Bird:init()
    -- load bird image from disk and assign its width and height
    self.image = love.graphics.newImage('bird.png')
    self.width = self.image:getWidth()
    self.height = self.image:getHeight()

    -- position bird in the middle of the screen
    self.x = VIRTUAL_WIDTH / 2 - (self.width / 2)
    self.y = VIRTUAL_HEIGHT / 2 - (self.height / 2)
end

self.image has a reference to an object of the class Image. So image:getWidth() is just a member function in the image class. The same applies to image:getHeight().

x and y (self.x and self.y respectively) coordinates are set so that the bird is exactly in the center. If you don’t understand the calculation used to determine it, please refer how Colton center aligns text in the last lecture.

The bird doesn’t move along the X axis at all during the game. It starts vertically and horizontally centered. It only moves along the Y axis during the game.

The Bird class also has a render method. Recall that delegating stuff to their respective classes can help reduce clutter in the main files, help us think more abstractly about the game and helps a lot with scaling.

bird3 “The Gravity Update”

Gravity’s forces an object to fall faster and faster (recall that it has an acceleration of 9.8 m/s^2). So the bird should fall faster and faster.

A Bird class is created and relevant functions refactored to it.

In Bird.lua he sets GRAVITY to 20. It’s a value that he found to work well. You can experiment and set any value. self.dy is the change in the Bird Y coordinate due to gravity.

function Bird:update(dt)
    -- apply gravity to velocity
    self.dy = self.dy + GRAVITY * dt

    -- apply current velocity to Y position
    self.y = self.y + self.dy
end

Notice in the code snippet above how GRAVITY is added to dy again and again. So dy i.e the bird’s movement becomes larger and larger as time progresses thus imitating the actual behaviour of gravity.

bird4 “The Anti-Gravity Update”

Now we try to add the jumping action once the space key is pressed. Intuitively you know that it should move in the opposite direction of gravity. dy should be negative (so that the bird moves up) and it should move up slower and slower as time progresses. By changing the sign of dy from positive to negative we can make it move in the opposite direction. If -5 gets the bird up by 5 pixels, -3 will get the bird up by 3 pixels. We want to slowly reduce the rate at which it moves up and eventually it should start moving down again.

This is done by adding small positive value to dy over time. See lines 29 - 36 in Bird.lua and run through the changes on a piece of paper for better understanding.

He also designs a way to refactor input handling to their respective classes so as to avoid clutter.

-- main.lua

-- line 70
    love.keyboard.keysPressed = {}

--line 77
function love.keypressed(key)
    -- add to our table of keys pressed this frame
    love.keyboard.keysPressed[key] = true

    if key == 'escape' then
        love.event.quit()
    end
end


-- line 98 - 110
function love.update(dt)
    -- other stuff here

    -- reset input table
    love.keyboard.keysPressed = {}
end

Line 70 in main.lua creates a global table to keep track of all the keys pressed. In Lua everything (except the basic data types) are tables. So love.keyboard.keyPressed simply adds a new field to the table love.keyboard.

Line 77 shows that every time a key is pressed the relevant key is set to true in the global table. We reset the table every frame (line 98 - 110). We wouldn’t want keyPresses to stick.

Now that we have a global table which has a record of all the keys pressed in that very frame, we can use it to query keypresses. Every class can look for keypresses it cares about and we can thus eliminate a large number of if conditions in main.lua. Example below.

function Bird:update(dt)
    -- apply gravity to velocity
    self.dy = self.dy + GRAVITY * dt

    -- wasPressed in just a helper function which access the
    -- global keyPressed table we just created and returns
    -- true or false
    if love.keyboard.wasPressed('space') then
        self.dy = -5
    end

    -- apply current velocity to Y position
    self.y = self.y + self.dy
end

Notice how the Bird class can simply check if space was pressed or not every frame and act accordingly. main.lua only updates the global table and individual classes can use love.keyboard.wasPressed (a function defined by us) to check if a key they care about was pressed in the last frame or not.

A nice in-depth write up on predicting the difficulty of Flappy Bird.

bird5 “The Infinite Pipe Update”

We need to keep destroying pipes as they leave the view. One could argue that we only use pointers (references) to the image object for every pipe we create and they wouldn’t amount to much. Colton explains that given enough time, we’ll run out of memory (and our game will definitely use more memory than required) and crash.

If you notice the init function in Bird.lua, it’ll become obvious that for every bird we put on the screen, it loads an image rather than pointing to a preloaded image object. It wasn’t a necessary design decision because we’re absolutely certain that we won’t load more than one bird into memory. The decision to share the image amongst all the pipes becomes necessary because there are infinite pipes.

local PIPE_SCROLL = -60 is set so that we can use it to smoothly move the pipe from the left to the right (recall that -60 is multiplied by dt).

Have a look at the Pipe:init(), you will notice that the X coordinate is just set outside the screen. The pipe exists but is outside the user’s view (recall that pipes don’t appear as soon you start the game).

Colton has used a long pipe sprite. By giving it different Y coordinates you can make it look like they have different heights. Notice in the picture below that all pipes have the same length but appear to be different to the user.

random.seed() is set in main.lua (it’s typicall a global thing which effects the random function as a whole hence we set it in the main file). A function to get the width of the pipe is created (will be helpful when we want to destroy pipes or check for scoring).

The Pipe:update() method is a single line function which updates it’s X coordinate. PIPE_SCROLL is the speed at which the pipe should scroll - self.x = self.x + PIPE_SCROLL * dt.

A table to keep track of the pipes currently in the screen is created in main.lua. We keep appending new pipe objects without specifying any key. It mimics the behaviour of a linked list.

A variable spawnTimer is initialised to keep track of the time elapsed since a new pipe was created. If it exceeds 2 seconds we create a new pipe and append it to the table.

function love.update(dt)
    -- line number 116
    -- dt is the time passed since the last frame
    -- if we keep doing this every frame
    -- we can get the actual time elapsed
    spawnTimer = spawnTimer + dt

    -- spawn a new Pipe if the timer is past 2 seconds
    if spawnTimer > 2 then
        table.insert(pipes, Pipe())
        print('Added new pipe!')
        spawnTimer = 0
    end

Now we have a table which keep storing all the pipes created. We iterate through all the pipes stored in the table and check if the right edge of the pipe (left corner coordinate + the width of the sprite) has crossed the left edge of the screen or not. If it has, we simple delete the pipe (see lines 129 - 136 in main.lua).

Also remember to have the right render order. If the pipes are rendered before the background image, the background image is simply drawn over the pipes and pipes remain hidden from view.

Check resources and check out the link on table iteration in Lua.

bird6 “The Pipe Pair Update”

A new class for clubbing pipes together is created. It uses components (pipes) to build something more complex.

The earlier table that keeps track of pipe pairs is renamed appropriately to pipePairs.

The variable local lastY hold the Y coordinate of the last pipe. We use this value while generating the next pipe. We want pipes that have gaps that are reasonably close to each other so that the game is playable. For this we need some information about the last pipe.

Line 132 of main.lua -> math.max() and math.min() are used to sandwich the Y coordinate of the pipe such that they are at least 10 pixels below the top edge of the screen and are at least 90 pixels away from the bottom edge of the screen. math.random() is used to reduce or increase the Y coordinate by upto 20 pixels with respect to the Y coordinate of the last pipe.

Tweak the y value to vary the level of difficulty or to make the pipes look more natural.

Now the Pipe class takes a string and a Y coordinate. This makes it easier to use the PipePair class. We can pass in lower or upper to invert the pipe.

Check line 26 of PipePair.lua to see how the lower pipe is rendered using the Y coordinate of the upper pipe. PipePair also has a member variable called remove that acts as a flag. If marked as true it is deleted from our table that keeps track of pipePairs. Read lines 143 - 157 of main.lua to see how PipePairs are updated and deleted. Do not delete elements from a table while iterating through it. It’ll result in indices being skipped.

Changes to the Pipe Class

init takes in the orientation (lower or upper).
Use of parameters to invert the image and set the Y coordinate of the pipe depending on the orientation of the pipe. See line 39 in Pipe.lua. We scale upper pipes by -1. Scaling by -1 gives a mirror image (hence upside down) but the size remains the same. Scaling by 2 doubles the size. Also notice how the Y coordinate is taken care of by using an if condition.

Interesting Overflow Error
If we let the bird fall for a while below the screen, it’ll suddenly appear from the top edge of the screen. Recall how it falls faster and faster (dy becomes very large) and then all of a sudden dy overflows and becomes very negative (negative dy results in an upward movement), one fine frame it renders above the screen and then gravity starts to act and it then falls from the top.

Stuff like speed of generation of pipes, gap height, lastY value etc can be tweaked to give a different feel and vary the difficulty of the game.

bird7 “The Collision Update”

local scrolling = true in main.lua is used to toggle pause and scroll states. If the bird collides with any of the pipes, it’s set to false and also pauses further rendering.

We iterate over the table of pipes to check if the bird collides with any of them or not (lines 152 - 157 in main.lua).

We use Axis Aligned Bounding Boxes just like last time. You might be wondering that the bird isn’t a rectangle and won’t entirely fill in the rectangle formed by it’s X and Y coordinates. You are right about it.

function Bird:collides(pipe)
    -- the 2's are left and top offsets
    -- the 4's are right and bottom offsets
    -- both offsets are used to shrink the bounding box to give the player
    -- a little bit of leeway with the collision
    if (self.x + 2) + (self.width - 4) >= pipe.x and self.x + 2 <= pipe.x + PIPE_WIDTH then
        if (self.y + 2) + (self.height - 4) >= pipe.y and self.y + 2 <= pipe.y + PIPE_HEIGHT then
            return true
        end
    end

    return false
end

We give an offset of 2 & 4 (see the code snippet above) so that we don’t frustrate the users. If we go by the actual X and Y coordinate, the bird will be close to the pipe (but not touch it) and yet the user would lose the game as the bird doesn’t reach the corner of the box formed by the X and Y coordinates. This would really frustrate the user. In addition we give in a couple of extra pixels so that it actually looks like it’s colliding. Also by giving a few extra pixels of space we are being more liberal to the end user and might actually make it more enjoyable for him.

Colton briefly describes an alternative style of refactoring. Instead of giving the bird a collides method, you can create a generic function which takes in two drawables and checks if they collide or not. It’s more scalable.

It is interesting to note that at times you might not observe a pixel perfect collision. The bird might be too much into the pipe. The collision function isn’t faulty nor was it’s execution delayed, the bird probably has moved too much in a single frame and we check for collisions after it has made it’s move (remember that movement, checking for collisions etc happens once every frame).

bird8 “The State Machine Update”

He showed the possible different states. Check slide#30 for the same. Certain input keys and conditions trigger transition from one state to another.

He imports a class called StateMachine, a base StateMachine class along with few game specific StateMachine classes. Also a few fonts are initialised.

gStateMachine is declared in Main.lua. It is a table with some mapping to States (each state contains functions that are relevant to it). It is a convention to prefix global variables with a g so that it’s instantly recognisable that it’s changing some global state (even though it doesn’t tell which file the global actually resides in). Likewise m denotes that it’s a member field of a class.

Now the love.update(dt) function in main.lua only directly updates code that is common to all states (the mountains and the ground keep scrolling irrespective of which state the game is in). It doesn’t make sense to duplicate this across all states and therefore remains in main.lua. Everything else is delegated to the respective function that gStateMachine calls. Similar is the case of the love.draw().

The StateMachine class is taken from the book mentioned earlier as optional reading in the lecture. StateMachine is a generic class and also a wrapper that we use render, update (position, health, score etc), switch and exit states (mostly for clean up. Graphic objects need to be destroyed etc). This is the class that actually takes in states like TitleScreenState, PlayState etc during it’s initialisation. It can be thought of controlling or managing the states in the way of calling their respective exit, render and update logic whenever needed.

BaseState initiates empty methods which can be inherited. It saves us from writing some broilerplate code for each state. TitleScreenState for instance inherits BaseState so it has the exact function definitions of BaseState and then some changes and additions are made to it’s functions. I would like to repeat:- TitleScreenState has it’s own member functions for cleaning, rendering etc but it’s controlled by the StateMachine class. The StateMachine class just handles possible states.

It’s like having a group of kids with different abilities (the state classes) and a teacher who takes charge of them (the StateMachine class).

bird9 “The Score Update”

A new score state is now included. Line number 96 of main.lua shows the relevant function call as well.

A new member variable scored in introduced in PipePair.lua. If the bird goes past the right edge of the pipe it is set to true. The score is checked by iterating through all the pairs of pipes (lines 53 - 61 of state/PlayState.lua). We check if the X coordinate of the bird has crossed the X coordinate of the pipe’s right edge (add the width of the pipe to the X coordinate of the left edge).

 -- simple collision between bird and all pipes in pairs
for k, pair in pairs(self.pipePairs) do
    for l, pipe in pairs(pair.pipes) do
        if self.bird:collides(pipe) then
            gStateMachine:change('score', {
                score = self.score
            })
        end
    end
end

If it collides with a pipe, we use gStateMachine:change() to switch from the play state to the score state which displays the final score. If you recall gStateMachine:change() can take in optional parameters as well. We pass in the score. By writing the gStateMachine:change() function in this fashion, we can pass in local variables while switching to another state. This saves us from creating global variables just for sharing data between two states. If we had to share a variable across all states or many states it would at times make sense to make the variable global. However only the PlayState and ScoreState need the score variable (no other state uses it), so it is better to keep your code base functional and pass in stuff required by the state by the way of a function.

Similarly we switch to the score state if the bird touches the ground (lines 92 - 96 from state/PlayState.lua).

bird10 "The Countdown Update"

It’s just another state. A new class CountDownState is created and a new key is added to gStateMachine which maps to the relevant function (look closely - the inline function actually returns an object). He uses hugeFont (initialised earlier) to display 3..2..1.

A few fields are added to the CountDownState class. count is used to keep track of the seconds pending until play and timer keeps track of the quantum of time elapsed since the last decrement in count. COUNTDOWN_TIME is the time between each tick, it’s set to 0.75 because 1 second between each tick is a little long.

Recall how we kept track of time to spawn pipes (kept adding dt through every frame) until it reached 2 seconds. A similar approach is used. self.timer = self.timer % COUNTDOWN_TIME is executed once the sum of the accumulated dts crosses 0.75. % or modulo is used to preserve information if the timer happens to exceed 0.75 (we would lose this information if we just set self.timer to 0), it starts the next tick with the excess time from the last tick. This helps keep a smooth track of time (if excess time is taking during a tick, we deduct that much in the next tick).

When self.count touches 0, we use gStateMachine to switch to the play state.

The title screen now enters the countdown state instead of activating the play state directly.

bird11 “The Audio Update”

We now have a table of sounds. They all map to data types that hold audio. The sound effects were created using Bfxr. He also used a free soundtrack from FreeSound (Please be sure to check the licensing terms before using them). He also sets one of the audio files to loop infinitely. Flappy Bird is an infinite game and the background music shouldn’t stop abruptly.

Sound effects are added in the respective places. For instance sounds earned for a point scored is places just below the code responsible for updating the score. In line 80 of PlayState.lua he layers two sounds on top of each other for the desired effect (quite common in games and sound engineering). sounds['explosion'] is white noise and sounds['hurt'] is a downward or sine wave type of sound.

Check resources if you’re curious about sound wave types.

bird12 “The Mouse Update”

It’s left as an exercise to the viewer. Use the function love.mousepressed(x, y, button). Remember how keypresses were made global to relieve main.lua. A similar approach can be used for mouse clicks as well. Each class can take care of the input keys that they care about.

You can find the code for same in the GitHub repository though.

End

GD50 Lecture 01 - Pong

Sun, 28 Jan 2018 00:00:00 GMT

Resources

Some Games Made Using Love - The list is quite interesting. If you feel Love and Lua aren’t worth your time, please do have a look.
What is VSync? - Read the first three paragraphs.
2 Min Video on Anti Aliasing.
Nearest Neighbour Interpolation - The Wikipedia article is actually quite understandable. This is used to create the box like retro feel.
State Machines - Amazing series on the “Theory of Computation”. The First 2-3 videos should suffice to get an intuitive feel of State Machines.
Table Syntax in Lua - Have a quick glance if you have the time.
Ternary Operator in Lua.
Simple Algorithm to Generate Random Numbers - If you are interested in getting a feel of how random numbers are generated. This will no way help you with this particular course though.

Lua

The lecture starts off with Colton talking a bit about Lua. It’s the Portuguese word for moon and was released in the year 1993 in Brazil. JavaScript / Python like language intended for use in larger applications. Especially popular in the game industry. It helped fasten development time by reducing compile time of large projects. Everything is a table in Lua. It’s very popular for storing data and code. I’ve personally seen many projects use Lua to write configuration files. In addition many in memory databases use Lua for stored procedures.

Love 2D

Love 2D is a game development framework written in C++ which uses Lua as it’s scripting language. Fast prototyping and support all platforms (even web and mobile).

Game Loops

It is essential to understand Game Loops. Games are an infinite loop which keep collecting user input (key presses, mouse movement, joystick movement), process them and render changed stuff on the screen. All this keeps happening as long as the user keeps playing. Link Mentioned in the Slides.

2D Coordinate System

2D Games use a 2D Coordinate System that is very similar to the one you are very familiar with. The origin is at the top left corner of the screen though and Y keep increasing as you keep going down.

Pong-0 “The Day-0 Update”

You can follow along the progress of the game by opening pong-0, pong-1 and so on in order. Small incremental changes are made in every update.

love.load()  -- This is used for initialising stuff like game state.

-- [[ Called every frame. dt is given the value since
-- the elapse of the last frame.]]
love.update(dt)

love.draw() -- called each frame. Use it to draw things.

-- Draw stuff on the screen. Can also align stuff
love.graphics.printf(text, x, y, [width], [align])


-- Window related stuff. fullscreen, vsync, window resizable or not
love.window.setMode(width, height, params)

-- Love 2D looks for a main.lua file to execute the game

function love.draw()
    love.graphics.printf(
        'Hello Pong!',          -- text to render
        0,                      -- starting X (0 since we're going to center it based on width)
        WINDOW_HEIGHT / 2 - 6,  -- starting Y (halfway down the screen)
        WINDOW_WIDTH,           -- number of pixels to center within (the entire screen here)
        'center')               -- alignment mode, can be 'center', 'left', or 'right'
end

6 is subtracted (line number 5 in the above snippet) to accommodate space (height) taken by the font i.e 12 units.

Check resources for VSync.

Pong-1 “The Low Res Update”

View Slide #21 for some more functions.

Steps are taken to give the font a retro like feel. I don’t understand how texture filtering algorithms work but in simple terms they decide how to scale textures. You have to choose the best filter for your needs. Textures like images lose quality and become blurry when zoomed in or may seem quirky when scaled down.

He showed a few texture filters and an appropriate filter that can give the desired retro fill.

They also start using a simple library called Push that takes care of scaling and resizing according to window dimensions. The official repository says push is a simple resolution-handling library that allows you to focus on making your game with a fixed resolution. You basically design the game for one screen size (called virtual width and height in the lecture) and push automatically scales stuff so that they look consistent on various screens. You can checkout the GitHub repository for more details.

Check the resources section and read about anti aliasing and nearest neighbour interpolation.

Here's some extra information about filters from an authoritative source.

nearest-neighbour filtering is always used to preserve pixelation when downscaling and upscaling textures. Information is often lost while downscaling but it’s always used to give those boxes (retro like feel).
bilinear filtering interpolates between pixels to create subpixels when upscaling particularly, resulting in blurriness.

Pong-2 “The Rectangle Update”

See slide #25 for the documentation of the functions used.

Fonts are immutable objects in Love 2D. So you have to load your font with respective sizes. You can’t use the same font with different sizes with one load. It probably loads the font with the specified size and stores it as some texture hence the need to load the font again even if you just want to change the size.

Every time you print something to the screen using printf, it uses the last set colour and font. It’s very similar to the way OpenGL operates.

Check the resources section and read about State Machines.

Pong-3 “The Paddle Update”

We poll to check if certain keys are held down every frame. It is important to understand key presses are not the same as holding the key. If the movement keys are held on we move the paddle by changing it’s X and Y positions. Also we need to store the X and Y coordinates of the paddle in variables.

PaddleSpeed is set to a value 200. To ensure that the speed of movement remains constant at all Frame Rates we don’t move it by a fixed value. Instead we want a smooth and even movement. So we multiple 200 by the dt (in fraction of a second) between each frame. Say our system is quite slow and the frame gets updated only 40 times every second so dt is 0.025 seconds. 0.025 * 200 gives 5. So the paddle ends up moving 5 units every frame.

You have to instantiate the same font with the sizes you want to use it. Say you want to use ‘Arial’ in the heading and also for smaller text you will have to load it twice with the respective sizes. Love 2D probably converts a font into some kind of texture once they are loaded. So remember that fonts are immutable in Love 2D.

Pong-4 “The Ball Update”

Pseudorandom generators are used to initialise the position of the ball. They normally use the time as the seed. Time is represented using a large number and therefore is a nice seed.

Math.min() and Math.max() is used to confine the paddle within the boundaries of the box. The position is updated either to the current position + delta movement or to the value where it touches the box. You can check lines 84 - 91 for the implementation.

Also the ball’s dx and dy can be changed randomly to deflect the ball at different angles.

A variable called gamestate is introduced to uniquely identify each state. For instance a key might do different things when the game is paused as against when the user is actually playing. An open menu, the pause screen, the victory screen are all examples of different states that a game can transition to.

Check the resources section and read about Pseudorandom Generators.

Pong-5 “The Class Update”

Classes:- Kind of similar in spirit to structs in C. Relevant variables are grouped together with associated functions. They help write modular code especially when things have an internal state to maintain. Bundles of data and code that are together. All variables related to a particular object (Ball, Paddle) and functions (move, collision detection etc) are refactored to their respective Classes.

The native way of using Classes in Lua is quite complex and isn’t similar to other programming languages that you might be familiar with. So they use a library called Class to create an abstraction that can enable you to use classes in a similar spirit.

Also it’s a convention in programming to capitalise the first letter of Class names and class related files for easy identification.

Pong-6 “The FPS Update”

The .. operator can be used to concatenate strings in Lua.

He starts displaying the number of frames our computer can render every second. It takes a second or two to calculate the value before displaying the correct FPS.

Pong-7 “The Collision Update”

Axis Aligned Bounding Boxes. It works when objects aren’t rotated and it’s an extremely simple collision detection algorithm. The axis of the object should perfectly align with our main axis. No edge of the desired box should exceed or cross the edge of the other box.

The ball accelerates as the game progresses. It’s done by the way of multiplying dx by 1.03 every time we detect a paddle collision.
When we detect a collision we abruptly shift the ball by 4-5 units in the other direction. Remember that we run 60 frames per second and in the future frames it might be possible that the paddle continues to collide, direction is changed again and this cycle might continue endlessly.
Change in dy value for randomness in angles.

Pong-8 “The Score Update”

It was skipped. He accidentally updated Pong-9 with the relevant code.

Pong-9 “The Serve Update”

Started keeping track of game score.
Thinking of games as State Machines are quite useful.
Can be used to thoroughly inspect, understand transitions and refactor the game at large. Also can be used to represent all actions and possible screens. Transitions between states happen when user gives an input (mouse click, keyboard presses etc).

Pong-10 “The Victory Update”

Score checking and transition to a new state called “done”. They use this state for distinguishing it from other states and thus enable different behaviour using if statements. Larger applications use modular ways to handle states. In the case of this small application they could do away with strings.

Pong-11 “The Audio Update”

Static assets like audio files can be stored in memory and played or they can be streamed from a file every time you want to play. Normally smaller files are loaded into memory during initialisation as opening files and streaming them is a slower process. For larger audio tracks it’s better to stream them as they can be many megabytes in size and might increase RAM usage unnecessarily.

You could even loop music so that it plays infinitely for cases like background music for the entire game or a particular level.

He also recommends using an application called Bfxr to create simple game audio.

Check the resources section and read about Lua Tables.

Pong-12 “The Resize Update”

Push takes in a texture and scales it. It needs a reference height and width to preserve aspect ratio. It also letter boxes (add some margin or padding) in case one of the dimensions is too large. This way you can avoid distorted aspect ratios and make your game look decent in all device sizes.

End

Introduction to Game Development GD50 - Notes and References

Fri, 26 Jan 2018 00:00:00 GMT

Notes, links and other resources that I find useful while taking this game development course. I always end up leaving courses midway and have decided to publish notes online for this particular course to build up some accountability.

I’ll try to get the notes online latest by 2-3 days post the release of the lecture. Life and laziness can get in the way at times. Update:- I’m far far away from this goal. Sorry about that.

I’ll also link to other material that I’ve found useful in the past (example lecture 1 doesn’t explain the algorithm for generating random numbers since it’s clearly outside the scope of the course, I’ll be linking to material that I’ve used in the past to understand the same)

At times I’ll probably skip the less interesting parts for the sake of brevity (for instance at times I might avoid bombarding you with documentation as it can be found in the course slides) and I’ll try highlighting the meaty bits.

How to Use this?

It can help use revise stuff. This is no way a substitute for the videos. Also you can use these notes to learn a little more about related concepts without having to Google for them yourself. I see myself using these notes after a couple of months as a quick way to refresh my memory without having to watch all the videos again.

Index

Please feel free to bring to my attention mistakes, other interesting resources and feedback using the comments section. I’m all ears and will do my best to incorporate them into these notes.

Novel DS & Algos Roundup #1

Sat, 04 Nov 2017 00:00:00 GMT

These are a few novel algorithms that I stumbled upon this year. I intend to curate and jot down stuff like this on a regular basis from now on so that I can quickly get to speed in the future (that is once I’ve forgotten about them). These also serve as notes. I don’t have to hunt down these links in the future.

Index

Bloom Filters
Music Shuffling
Rsync
Hyperloglog
Pratt Parsing

Bloom Filters

It’s a space efficient algorithm that checks if a particular item exists in a list or not. It trades memory requirements for accuracy. It has a very high accuracy but isn’t right all the time (you can tune it to use more memory and increase accuracy or reduce memory requirement and accuracy at the same time). Ingenious use of hashing.

Bloom Filter for Dummies
This is a well explained article keeping a sample usage in mind. Loved it.
Tutorial with a Demo

Music Shuffling

Random algorithms obviously aren’t deterministic. When using them it’s possible that at times you might end up the same number multiple times clustered together. For example 1, 2, 2, 8, 4, 5, 5, 5. This algorithm ensures that the songs are uniformly distributed.

The Art of Music Shuffling
The above link is visually explained and is very easy to grok.

Rsync

Rsync efficiently transfers only the differences between the two files. Say you have an outdated file A on your server and the latest file A on your personal computer. You have to replace the file on the server with the latest file. The naive way to send the file would be to send the entire file from your computer to the server. It uses more bandwidth and takes more time.

The smarter way would be to just send the differences between both the files. Only the parts that have actually changed. Rsync uses hashing to find out the differences and can then enable you to send only the differences saving you time and bandwidth.

Overview of How Rsync Works

Hyperloglog

I stumbled upon this during my attempt to read the source code of Redis. It is used to return the number of distinct elements in a list or container. It uses lesser memory than the traditional approach (iterate over each thing, if not already encountered increase count by 1) but isn’t exact. It’s pretty accurate that even Redis uses it.

Explanation by Antirez
StackOverflow
Points to a number of related resources and gives a decent explanation as well.

Pratt Parsing

Pratt parsing is a simple parsing technique. I haven’t worked with parsers so I don’t know the benefits it offers over other parsing techniques.

These tutorials go really in depth.

Redis Source Code - DS and Algos - adlist

Sat, 04 Nov 2017 00:00:00 GMT

If you are not aware of the main article, please read it first - Index of Deep Dive into the Source Code of Redis.

Please keep the source code of Redis handy. Also have search commands for your respective text editors handy. You might have to perform quite a few searches :)

Meta Info

Difficulty - Very Easy
Should be Familiar with -
- Doubly Linked Lists
- Pointers in C (specially void*)

adlist.c & .h

This file is quite easy to understand. Perhaps you could try it out yourself even if you consider yourself a noob.

It’s a doubly linked list which allows you to store anything. It uses void* in the value field to point to anything. Structures used - listNode, listIter and list.
The node struct is a single unit of a list. It has a pointer to next, prev and a void* pointer so that it can store anything.
The listIter struct is used to create a neat iterator (cleaner that what you normally would do in C but not as clean as the ones in languages like Python or JavaScript). It’s handy to access the next element, duplicate the list etc. Directions are defined as 0 & 1. 0 goes forward from the head to the tail and vice versa. I’ll explain a bit more about it below.
The list struct has three function pointers (free, dup and match). I’ll explain their uses shortly.
Macros are used to access the elements inside structs. For instance it’s easier to get the length using a function rather than using a pointer.

/*
 * Remember macros are substituted
 * before compilation. So writing a function is actually
 * less efficient (forget over optimisation for a while) than a macro
 * And why not make use of the elegant features that C has to offer?
 */
#define listLength(l) ((l)->len)

// Cleaner as people prefer using
// functions or member variables rather than pointers
listLength(l) // cleaner than (l)->len

free, match & dup
If you head over to the code you will notice that the node structure’s value parameter is a void *pointer. It means it can point to any value under the sun. You could point it to a struct that represents a client who just sent a request. Say you’re done and want to free the entire doubly linked list. If it were a simple int, just freeing every node in the doubly linked list would do. Remember the node itself is a dynamically allocated object and you have to free it. The value parameter might also point to another dynamically allocated object. The client struct might have arrays for IP, the actual request and so on which need to be freed.

Say now you decide that your doubly linked list should hold dog kennels. It probably has an additional pointer to a dog struct which also should be freed. The freeing mechanism therefore differs for each struct. To generalise the library so that it accommodate any use case you can pass in your custom function to free. It’ll be called for every node when you free the list.

The same logic goes for match and duplicate. ints and chars can be directly compared but what about pointers? What if you want to compare two identical book names but different pointers point to them. A simple == check will return false as the pointers are not equivalent. You would have to follow the pointer and compare the entire string. So you can pass in a custom function that will be called when you have to compare values in the doubly linked list.

I’ll just skip over the very obvious functions. The explanation given above should be sufficient to understand them.

Some things worth noting. Might be useful in case you decide to write a doubly linked list or design an API ;)

//After is an additional parameter that specifies if the node is to be
// after or before the position of *old_node
listInsertNode(list* list, listNode *old_node, void *value, int after);

Nothing fancy but it’s a simple example of using the same function to do many things by modifying the flags you pass in.

// Returns a pointer to the head or tail
// You don't have to use list->next or list->prev to access elements
// one by one. You can pass in an itertor and keep calling next to
// access all the elements of the list. Imagine the ease.
listIter *listGetIterator(list *list, int direction);
listIter *listNext(listIter *iter);

// Say I want to empty a list.
// Without the abstraction above I'd have to

// for forward iteration
for (listNode *temp = list->head; temp != NULL; temp = temp->next) {
    // some operation to perform
}
// for backward iteration
for (listNode *temp = list->tail; temp != NULL; temp = temp->prev) {
    // some operation to perform
    // Example duplicating or emptying a list
}

// With the abstraction above
// 0 is used to set the direction
// either head to tail or tail to head
listIter *studentsIter = listGetIterator(studentList, 0);
while ((node = listNext(studenIter)) != NULL) {
    // some operation
}

If you are not able to appreciate the improvement, let me jump in.

Read the two for loops in the code snippet above. Look the number of times I have to deal with the pointer. I also have to increment it. Say I perform such operations 20 times, I’d be writing this for loop 20 times. I could accidentally interchange head and tail while copying and pasting stuff.
The for loop is my style of iterating over a doubly linked list. You might prefer a do-while loop for the same. Such small differences can crop up and clutter the mind of the person reading the program. So anyone who wants to iterate over the list should use the function rather than their own style.
It thus is common practice to create functions for things even if they are trivial or don’t reduce the number of lines.
Another thing to notice. By giving it a name it becomes clearer as to what’s being done. To best elaborate imagine a complex single line statement (eg regex matching after string concatenation and slicing). Wouldn’t it be better to just make a function rather than have the programmer try understanding the various parts of that single line.

// Provides an option to get element at a certain position like how
// you'd do in a normal array. Also supports negative indices found
// in languages like Python. Nothing fancy in this function but I like
// the way he handles negative indices.
// -1 returns the last element, -2 the second last and so on
listNode *listIndex(list *list, long index) {
    listNode *n;

    if (index < 0) {
        index = (-index)-1;
        n = list->tail;
        while(index-- && n) n = n->prev;
    } else {
        n = list->head;
        while(index-- && n) n = n->next;
    }
    return n;
};

Deep Dive into the Source Code of Redis

Fri, 03 Nov 2017 00:00:00 GMT

This is a work in progress. I’ll keep updating this article on a regular basis.

This is my attempt to thoroughly understand and document the code base of Redis. I’ve used actual file names which Redis uses and have split them into categories which I think they best fit.

DS and Algos

adlist
A generic doubly linked list
ziplist
A memory efficient doubly linked list for integers and strings
zipmap
A memory efficient but slow data structure that maps a string to another string

Some Major Functionality

ae
An event driven library to concurrently perform operations like network requests etc.

Helper Libraries

Processes Links

Good to Know

header guards

Flow Once You Start Redis

Google File System Summarised (Part 3)

Tue, 31 Oct 2017 00:00:00 GMT

Link to Part 1:- Google File System Summarised Part 1
Link to Part 2:- Google File System Summarised Part 2

System Interactions

Mutations obviously have to be performed on the all the replicas. Say a client wants to mutate a file. Amongst the many replicas avai lable, the master grants a lease (think of it as permission to be modified). They call this particular replica the Primary.
The lease (permission) times out in 60 seconds. It however keeps getting extended as long as a client is mutating it.
The primary decides the order of mutations that are to be applied. All the other replicas follow it.
Control and data flow are decoupled. All the replicas follow the primary for the order of mutation. Data is stored in each chunkserver’s buffer and used when the time comes. Process of data transfer is explained below.
The client gets the location of the primary and all the replicas. It caches this information to reduce the involvement of the master. - Data is passed to all the replicas. A scheme that exploits network topology is used rather than what you’d normally expect (i.e passing to the primary and then to the replicas). Also each server uses it’s full network bandwidth to pass data to the closest replica rather than split it’s capacity by passing to multiple servers simultaneously. Say the client wants to push to 4 chunkservers (S1 to S4). The closest server to the client is S1. The client uses all it’s bandwidth to pass data to S1. Closest to S1 is S2. S1 uses all it’s capacity rather than splitting transfer rate between S2 and S3. They also minimise delays by pipelining the transfer over TCP. Also a chunkserver forwards as it keeps receiving (it does not wait for the entire data) to further minimise delays. They have a setup which enables them to measure distances between chunkservers using IP Addresses.
When all the replicas have received the data, the client then sends a write request to the primary. It gives a serial ID to every mutation (possibly from multiple clients). Also remember that it already has the data in it’s internal buffer. The primary forwards the requests to all the replicas (the primary has executed all the mutation requests by now). They mimic the order followed by the primary.
The primary replies to the client. Errors encountered at the replicas are reported to the client. The mutation will not fail at the primary as it only forwards requests after it’s done executing them. The client retries a few times. The details like does it retry only the failed chunks or all the chunks again etc aren’t delved into in the paper.
Atomic Append Records (appending at least once at the end of a file if you recall from the last part) follows a very similar process. However unlike a new file it cannot be written at an offset which the client chooses. The primary replica chooses the offset at which the file should be appended (if many clients write to the same file the offsets are more prone to clash and create undefined regions). There’s an additional check that happens. If appending the new data causes the chunk to exceed 64 MB, it pads the current chunk to take 64 MB and retries the append in the next chunk. Append sizes cannot exceed 1/4th the size of the file to keep such fragmentation in control. GFS only guarantees that data will be written at least once but each replica of a chunk might not be identicaly byte by byte (I’m a little fuzzy about this but I think differences are caused by different amounts of padding used or data being appended more than once on particular chunkservers).

Snapshots

Copies a file or directory almost instantaneously.
Standard copy-on-write techniques used. Read more about it on Wikipedia.
Used while creating copies or as a precautionary measure while creating checkpoints.
All outstanding leases are revoked. It prevents mutations to the file(s) during the process.
The newly created snapshot points to the same chunks as the source file (remember copy-on-write? To save resources a reference to the particular chunk is used rather than creating a duplicate copy). Actual copies or deep copies are only made when clients make a request to modify the chunks.
On the first write operation to that chunk post the snapshot, the master notices that it has a reference count greater than one. Reference count is used to keep track of the number of shadow copies or references(as explained in the previous point) that exist of the particular chunk. Since it’s a mutation, the references should now be turned into copies as the original probably should not be touched (they are part of a snapshot if you recall). The master tries to optimise this process as well. Instead of sending the chunk over the network and creating copies across chunkservers, it requests chunkservers that already have that particular chunk stored locally to create another copy. It grants a lease to one of these chunkservers. After all this is done it sends the client information about the primary.

Master Operations

Namespace Management and Locking

Master needs to acquire locks for some of it’s operations. Example while taking a snapshot.
Efficient locking mechanism over directory and files so that many concurrent operations can be performed.
The master does not have a per directory data strucuture that lists the content of that directory. It just maps fullpathnames to metadata (prefixes are compressed to reduce RAM usage by metadata).
So /home/user/data.dat is actually a key with some metadata as value. There are no directories as such in GFS. The metdata from the key /home/user/data.dat points to relevant locations. Reading these locations enables the client to get the file.
Each opeartion requires various locks. For example /d1/d2.../dn/leaf (leaf indicates the final directory or file it is performing an operation on) it will acquire read locks on /d1, /d1/d2… /d1/d2.../dn and a writers locks on /d1/d2.../dn/leaf. Getting a writers lock allows writing to /d1..../dn/leaf and a reader’s lock on the remaining directories prevent /d1, /d1/d2 etc from being deleted or renamed. To my understanding directory /d1 indicates all keys (fullpath file names) which start with /d1. Given full paths to files, if two filepaths start with /d1 they belong to the same directory.
They give an example of the locking system prevents a file being created in a directory which is being snapshotted. I’m not sure if my understanding is correct, so it’d be better if I skip it. I couldn’t find any other link online which explained this particular section in depth.
Multiple file creations can happen in the same directory as well. Both the writers acquire a read lock on the directory (the directory is protected from being renamed, deleted or snapshotted) and write locks on the files being written (this serialises attempts to create a file with the same name).
Read-write lock objects are lazily allocated and deleted. They are acquired by level in the namespace tree and then lexicographically within the same level. Doing this (shallow to deep) prevents deadlocks.

Replica Placement

Physical placement of servers across racks.
Make sure replicas exist in different racks to maximise data availability and integrity. Say an entire rack is down due to an outage clients can still read and write.
They explain that bandwidth in and out of a rack maybe less than the sum of the bandwidths of individual machines. Hence placing it in various racks can help clients exploit reads from various racks. In the case of mutations where the client has to send data, multiple racks actually are disadvantageous as data has to travel longer distances. It’s a tradeoff that they are quite satisfied about.

Creation, Re-Replication, Rebalancing

Creation and Re-Replication

The goals of a master are to place replicas on servers with less than average disk utilisation, spread replicas across racks and reduce the number of recent “creation or writes” to each server (even though writes are cheap they are accessed by many clients simultaneously even while being written (recall checkpointing by writers and readers) which might create additional load).

Chunks need to be re-replicated if the number of copies fall (data corruption on a server or a replica is unavailable) below the threshold (normally it’s set to 3 but it’s configurable as mentioned earlier). Instead of re-replicating all of them at once the master prioritises re-replication to prevent these cloning operations from becoming bottlenecks. Restrictions are placed on the bandwidth of each server for re-replication so that client requests aren’t compromised.

How are Re-replications Prioritised?

A chunk which is two short of desired replicas is given precedence over a chunk which is short by one.
Live files rather than recently deleted files (discussed in the Garbage Collection section). Files that are marked as deleted are just renamed temporarily and deleted only after a few days. So replicas of them can exist for a few days as well.

Rebalancing

Master also rebalances regularly to achieve the above said goals. It may move replicas form one rack to another, try to bring disk usage in a server closer to the average and so on. Any new chunkserver is filled up gradually by the master rather than flood it with loads of write operations.

Garbage Collection

Any file deletion is logged immediately but the resource isn’t deleted immediately. It is renamed to some hidden folder. The master regularly scans the filesystem and deletes such files if they are older than 3 days. It also deletes metadata of chunks that have no files (it can check its file to chunk mapping and find out chunkservers that are not used) or only contains deleted files. Also during exchange of HeartBeat messages the master can instruct other chunkservers to delete replicas of the type of chunks described above.

Advantages Over Eager Deletion

Bookkeeping is reduced. Master does not keep track of messages lost (and hence keep retrying) in an environment where component failures are common.
Done in batches when load is less.
Tagged along with regular activities like namespace scanning.
Safety net against accidental deletions.

Disadvantages

Storage isn’t available immediately. Poses problems when storage space is scarce. (It is possible to reduce the period to trigger deletion from 3 days to a lower value)
Applications that frequently create and delete files might put the system under stress (storage is used immediately but freeing can take 3 or so days). To overcome all this the clients can specify directories that are to be stored without replication. They can also specify directories where deletion takes place immediately.

Stale Replication Detection

The master increments the chunk version everytime it grants a lease. After the replicas of this chunk make mutations they are updated to the current version as well.
If the replica isn’t available during mutation it would miss the mutation and hence would show an outdated version number. During regular garbage collection the master can take care of such replicas.
Additionally the master sends the version number to a client while it requests a mutation and also while a chunkserver clones another chunkserver.

I haven’t covered Section 5 (Fault Tolerance and Diagnosis). Mostly because the post is already pretty long and also I’m lazy. Moreover Section 5 is only around a page long and can easily be grokked provided you have a reasonable idea of what I covered in these 3 parts.

Review of the Book "i want 2 do project. tell me wat 2 do."

Fri, 20 Oct 2017 00:00:00 GMT

This is a short review of a book that I read last year.

Summary for people in a hurry:-

Just out of College? - I’d definitely recommend reading the book end to end.
Reasonable Experience? - Skim through everything and drill down on areas of your choice. You’ll pick up a few neat things you weren’t aware of.
Veteran? - Reading it end to end won’t be good use of your time. I’d recommend selective choosing and going through the reference section (less than 1 page for every topic covered) for recommendations (books, blogs etc). They are pretty interesting and will help you delve deeper.

At first when I saw the photo of Shakthi Kannan with the book on his website I thought that he was holding some sort of banner and was looking for projects to work on.

It’s an excellent read for people who know how to code but are yet to digest the fine details of the trade. College students with no work experience are a perfect example. Shakthi covers a wide plethora of information (really wide) that programmers are expected to know. People who’ve learned Computer Science at home from books / online tutorials should definitely read it. I strongly feel it fills certain gaps that help smoothen your journey as a programmer.

I love the quotes that he mentions in the book. I’ve started following many of the authors of those quotes. The details that he stresses on are useful. For instance I remember him mentioning to be careful while posting to mailing lists as the mails are often stored permanently on those servers (I could instantly relate when I saw a few substandard emails and later learned that they were hosted on some email listing site). I’ve also understood the importance of a few things that he mentions months later when I encountered them at my job. I particularly like the resource section of the book, it helps me quickly glance things I want to read more about. He has also explained things quite well using real life examples. Many of the books / resources mentioned in the book are spot on (I’ve read many of the books he mentioned before reading his book and I can personally vouch for them). He doesn’t leave out health and sanity. I was under the impression that RSI was a distant possibility (I was proved wrong again. People who I know very well have suffered in the past). I’ve realised that I am going to face more problems than I care to admit and on the bright side there are always going to be helpful sources that I haven’t even considered. Many of the tips that he mentions will not be applicable to you at this very moment or help you solve your current problem but you’ll find them useful sooner or later.

If you’ve had even a little experience with online communities, have attended a few local meetups or have worked for sometime in a real work environment you might be aware of many things that he discusses in the book. You would still benefit if you glanced through a few sections of your interest. I am confident that you will stumble upon a few tips / tools that you wouldn’t have been aware of.

If you’re a veteran and are involved in open source development from quite some time, this book isn’t for you. You’d be better off researching specific areas that you think you need to improve / streamline.

Google File System Summarised (Part 2)

Fri, 13 Oct 2017 00:00:00 GMT

Link to Part 1:- Google File System Summarised Part 1
Link to Part 3:- Google File System Summarised Part 3

I’ve tried matching the sections and the content appearing below them as closely as possible to the original paper. However I’ve shuffled sections in a few instances for the sake of brevity and ease of understanding.

Metadata

Three types of Metadata. The first two are persistent and are also replicated on remote machines. All mutations to the first two are logged to an operation log

file-namespaces
file to chunk mapping
location of chunk replicas (not persistent)

In-Memory Data Structures

Stored in memory thus fast.
Periodic scan of state made possible because it’s efficient (enables garbage collection and re-replication).
Approx 64 bytes of data for each 64 MB chunk.
Drawback - Master limited by the amount of RAM. But it’s easily overcome by adding more RAM.

Chunk Locations

No persistent record of chunks that hold the replicas. They’ve reasoned that the chunkserver ultimately decides if it has that chunk or not. The portion of the disk might have gone bad or might have vanished for some reason. Hence a persistent model only makes things harder as everytime the chunks in a chunkserver changes the master has to be notified and updated about the same. With hundreds of servers it’d contribute considerably to technical debt.
Master collects this data during startup or whenever a new chunkserver joins.
Master can monitor changes with the heartBeat message it sends.

Operation Log

Record of important metadata changes.
Helps in defining the order of concurrent changes. Maintains a logical timeline.
Files and chunks and their versions are identified by the logical times at which they were created. (Master maintains a version number for each chunk to identify up-to-date chunks and stale ones. Whenever the master gives the client permission to append, it updates the version number and informs all up-to-date replicas as well. So if a chunk server is down, it misses the version update. When the failed chunkserver is up again, the master can garbage collect that particular chunk as it’s an older version.)
Should be stored reliably and only shown to clients when the change is persistent. Since it’s cardinal, it is replicated across machines. Logs are batched before being written to disk on the local master as well as remote machines to reduce burden.
The master can restore it’s state by replaying these logs. Even during re-replication when logs are read from disk, they are batched to reduce it’s impact.
When logs size increased beyond a threshold, the master creates a checkpoint (it’s in the form of a binary tree that maps files to chunks and requires no parsing) and stores it to disk.
Incoming requests to mutate files should not be blocked while the master is creating a checkpoint. To tackle this the master switches to a new log file in a separate thread. Creating such a checkpoint takes a minute or so for a cluster with a few million lines. This new checkpoint includes all the mutations before it. (I’m not sure as to how it’s done)
Recovery only needs the latest complete checkpoint and associated log files. Older checkpoints and log files can be deleted (though Google keeps them safe in case of disasters).
The code written to help the master recover after failure detects and skips incomplete checkpoints (so failure during checkpointing is taken care of).

Consistency Model

As mentioned earlier weak consistency (different clients may see different data for small periods of time. They’ve ensured that it doesn’t drastically effect the clients though.) for performance and simplicity.

Let’s start by explaining a few terms for those who aren’t familiar with GFS. The terms consistent, defined, undefined, inconsistent defined a file region or chunk in this context.

Consistent - A file region is consistent if all the clients see the same data no matter which chunkserver they read from.
Defined - It is a region after a file mutation. The client can see the mutation in it’s entirety. Say one client wants to append data and there is no other client appending to that chunk, it is defined. By implication defined also means consistent.
Undefined - When many clients mutate a region successfully, it is not possible to see what one particular client has written. It however is consistent.
Failed concurrent writes leads to chunks that are undefined and inconsistent.

Mutations are either writes (new files) or appends to existing files.

Mutations are applied in the same order to all the replicas.
Stale replicas do not participate in mutations (recall version numbers for each chunk).
GFS lets a client write new data at a location that the client seems fit. However clients cannot append to a file as per their wish. GFS gives them a file offset to write to (GFS may include some padding to take care of concurrent writes to the file).
They’ve also mentioned that some data might get corrupted (many writers writing to the same region) in the process but the amount of data which can get corrupted (inconsistent as explained earlier) pales in comparison to the total data. Applications (normally work on the entire data set) can still work uneffected.

How are data inconsistencies dealt with?

If you recall clients cache chunk location data. So it might read data from a stale replica. When this happens agreed that it doesn’t get the complete file but it doesn’t get outdated or wrong data either. Moreover the cache timeout limits the possibility. When the cache expires and the master is contacted again it gets updated info.
Applications (i.e clients) can distinguish between defined and undefined regions.
GFS identifies stale servers by handshaking (recall heartBeats) and ensures data integrity by checksumming.

Also keep in mind that the clients have to accommodate to ensure the overall success of the Distributed File System.

Mutations only by appending. No overwriting of existing data in chunks.
If the writer(client writing to the filesystem) generates a new file, it renames it to something permanent after it’s done or periodically checkpoints it. These checkpoints can also include checksum or hashes.
The readers (clients reading the data) only read data upto the latest checkpoint.
Client side checkpointing also ensures that readers don’t read data that writers feel are incomplete. Say the writer creates a checkpoint (indicates that it is done writing that portion) and then starts appending (still in progress), the reader only reads upto the checkpoint.
The clients have to deal with duplicates (as GFS promises to append at least once, so it might append a record more than once) and padding (GFS may or may not let a writer append immediately after file end). The writers have to write some extra information (special bits or patterns) to indicate the start of an actual record. This helps avoid padding. Unique identifiers can be used for each record so when readers access the data they can avoid the identifiers they’ve already encountered.

This part would have given you a hint of what happens behind the hood. I’ll cover more details about the flow in the next part.

Awesome Computer Graphics Resources

Sat, 07 Oct 2017 00:00:00 GMT

Please keep in mind that I haven’t even started most of them. I find the links mentioned here to be great learning material and would eventually like to work my way through them. So I’m not in a position to prescribe learning paths.

Grok the Graphics Pipeline

An Intro to Modern OpenGL
Vertex Buffer Objects
Shaders
Same blog as the one mentioned above. Anton has other great stuff as well.
Graphics Pipeline Explained in Great Detail

Some Simple Tutorials

SDL Tutorials by Lazy Foo
SDL is quite popular. It’s a cross platform game development library.
Lighthouse 3D Tutorials
LearnOpenGL
Another great set of tutorials written in a fashion that resembles a book. Have stumbled upon this site many times.
50 Small Incremental Tutorials
Learning Modern 3D Graphics Programming
Written in the form of a tiny book. Well written. Tops in my list of todos.
Another Great Set Written as a Book
Again not exactly a tutorial but more like a book. By the comments it has garnered I wager that it’s awesome.

Hand Held Projects

Create a Roguelike Game in C#
Create a Complete Professional Quality Game
Looks very promising. It’s composed of videos and I think it’ll be of great help to people who want to work in the gaming industry.

Inspiration

Loads of User Submitted Shaders
JavaScript Gamer Under 13kb
Super cool concept. You have to build a game using JavaScript under 13kb (assets, code everything!). The kind of stuff that they’ve build is unbelievable. The source code for most of them are easily available as well.

Blogs

Real Time Rendering

Notes

Google File System Summarised (Part 1)

Thu, 05 Oct 2017 00:00:00 GMT

Paper:- Google File System
Link to Part 2:- Google File System Summarised Part 2
Link to Part 3:- Google File System Summarised Part 3

The Google File System has inspired many Distributed File Systems (Big Table and HDFS to name a few). It’s interesting to note that GFS gives up strong consistency (it promises to be eventually consistent) for the sake of simplicity and performance. Also GFS was developed for certain kinds of applications (example mapreduce) and IS NOT suitable for most applications. Moreover Google has replaced GFS with Colossus as a response to changing needs.

Need Case and Assumptions

Hundreds or thousands of inexpensive computers accessed by a large number of clients. One Master Server. Components fail often.
Multi GB files are common. Millions of 100MB sized files. Each file contains application objects like web documents (my understanding is that the files are similar to zipped files. One large file containing many small files). Arguments for large files
- Fewer large sized files are better than billions of Kilobyte sized files (storing metadata and file mappings becomes a pain as we will see later on)
- Redesign IO operations and block sizes due to the specific needs of the applications
Most of the file mutations append data. Random writes in a file is almost zero.
Once a file is written, it is mostly read. Most of the times the reads are sequential. Surprisingly a large number of applications can benefit from such an assumption. Data analysis applications passing through large chunks of data, mapreduce applications reading large files during reduction stage etc. Therefore there are two types of read operations.
- Large Streaming Reads - Reads 100s of KBs of data sequentially.
- Random Reads - Reads a few KBs at some offset of a file.
Applications & GFS have to complement each other’s design. This increase flexibility.
- Normally clients requesting data cache it. Clients using GFS should not cache it as the data is large and might be subject to change. (Clients only cache metadata like location to a particular file on a chunk)

Interface

Familiar but does not implement POSIX.
Create, Read, Open, Delete.
Snapshot and Record Append.
Record Append supports simultaneous appends from multiple clients without blocking.

Architecture

1 master many chunkservers
Chunkservers
- Uniquely identified by 64 bit integers
- Store files on local disk as Linux files
- Replicated. Normally 3 times (can be configured)
- They take care of reading and writing data. The master gives the client the chunkserver location and the file byte range.
Master
- HeartBeat messages - Periodic communication with chunkservers. Gives instructions and finds out more about their state.
- Metadata (map individual file to a chunk and byte range, location of each individual chunk, replicas store etc)

Single Master

Simplified actions when it comes to file mapping and replication. Single leaders and masters are known to reduce complexity (Eg Raft Consensus Algorithm).
Minimum involvement with the clients. Say the client wants to read or write to a file, it gets the location from the master and deals with that chunk server. Also this metadata (i.e server location (with details of their replicas( and byte offset for a particular file) is cached. So the client needn’t bother the master frequently for metadata.

Chunk or Block Size

Key design parameter. They’ve designed each block size to use 64MB. To help you contrast :- A normal linux system uses 4KB as it’s block size.
Larger block size reduces client master interaction. It can get the required details about file locations in one request. It is not uncommon for clients to cache metadata for multi TB datasets hence even random reads can be performed without the involvement of the master.
Lazy space allocation is used as a counter measure to internal fragmentation. Read this StackOverflow question for more details. You can also read about Sparse Files.
Since clients are probably going to read data from a chunkserver for a long period of time (remember the base assumption, files are large and data is read sequentially), it can make use of a persistent TCP connection.
Small files can turn the chunkserver which house them into hot spots (active with lots of incoming requests). This problem can be alleviated by having more replicas of such files. This StackOverflow discussion might be of use. The requests are therefore distributed over a larger number of servers.
They had an executable hosted (it fit in a single block). It was used by hundreds of machine on startup at the same time and this created problems. They solved this issue by increasing the replication factor and staggering the application start times (instead of starting all the applications that needed the file at once, they spread it out to reduce the number of concurrent requests). This is another such occassion where they’ve resorted to making modifications to the client to keep the architecture simple.

I’ll cover the remaining sections in the future :)

Awesome Distributed Systems Resources

Wed, 27 Sep 2017 00:00:00 GMT

New to the Concept of Awesome Lists?

They are a curation of exciting resources on a particular subject. They are actually quite popular these days.

This list would have been incomplete without the input of Shrayas. I’ll keep adding stuff as and when something catches my eye. Please keep in mind that I haven’t personally gone through most of them. Some of them are recommendations by people I know and I’ve actually read a few of these resources.

Blog Posts and Links

Notes on Distributed Systems for Young Bloods
Plain English Introduction to CAP Theorem
Understanding RPCs
CA CP and AP
Read this StackOverflow answer after you’ve read the Plain English Introduction to CAP Theorem.
Why CA systems can’t exist?
High Scalability
An awesome blog with loads of resources on Distributed Systems.

Papers

Time, Clocks and the Ordering of Events in a Distributed System
This is the most cited paper in computer science.
MapReduce: Simplified Data Processing on Large Clusters
A novel way developed at Google to abstract away problems concerning distributed systems from the end programmer.
In Search of an Understandable Consensus Algorithm
A relatively easy to understand consensus algorithm. I’ve written notes for this paper here.
Paxos Made Simple
Paxos (a consensus algorithm which people consider pretty hard to understand) explained in simpler terms.
Google File System
Distributed file system at Google. It is 14 years older than this blog post. Google has made a quite a few changes to their file system in light of recent developments. I’ve written notes for this paper here.

People to Follow

Aphyr He runs Jepsens. Jepsens focuses on the safety of distrbuted systems and puts their claims to a test. He trains people as well. Notes from his 2 day training. They’ve also analysed various leading databases and queues. You can find the reports here.

Quick Hacks 1

Tue, 26 Sep 2017 00:00:00 GMT

Here are a few ways of solving problems that deviate from the norm. They aren’t the best solutions nor do I recommend you to use these in production or critical systems. These may or may not be elegant or well expressed.

I think these are quite creative. I intend to document more of these as I find (and hopefully create in the future) them. I wager that they are a nice gentle examples of thinking outside the box. Also these are examples of how one tool can be used to solve a wide variety of problems.

1. Removing Duplicates and Sort

I don’t recall where I saw this (It was some Go repository). Now languages like Python, Haskell and JavaScript have inbuilt functions to remove duplicates from lists or arrays. Go does not.

You can quickly whip up an one liner in Python. Let’s see how this problem was solved in Go.

I feel this is a neat hack as you don’t have to implement a function to remove duplicates. The guy basically used the sort function and turned it to a tool to help him remove duplicates.

Problem:- It’s a slice (array, list whatever) of Strings. Go has a function to sort a slice of strings but no inbuilt function to remove duplicates.

import (
    "strings"
    "sort"
)

// a function which takes in a list of strings and returns a list of strings
func SortAndRemoveDuplicates(badList [] string) [] string {
    var lastIteration string

    // a variable to hold the return value
    var finalList []string = make([]string, 0)

    // badList is now sorted but still has duplicates
    sort.Strings(badList)

    // we iterate over the sorted list
    for _, individualString := range badList {

        // since it's sorted same strings appear together
        // if the last added string to the final list it is a duplicate
        // we only add it to the new list if the current string is not
        // identical to the previous string that was added

        if individualString != lastIteration {
            finalList = append(finalList, individualString)
            lastIteration = individualString
        }
    }


    return finalList
}

2. Accessing Elements in an Array

I saw this in a competitive programming repository.

Problem:- Access two consecutive elements in an array. If it’s the last element access the first element. First and second, second and third and so on until last and first. The length of the list is known in advance.

Note:- I think reduce (the functional programming pattern) is a good fit for an elegant solution to this problem. However I haven’t tried implementing it.

// normal straightforward way
var myList = [1, 2, 3, 4, 5, 6, 7, 8];
var indexOfLast = 8;
for (var i = 0; i < myList.Length; i++) {
  if (i == 8) {
    console.log(myList[i], myList[0]);
  } else {
    console.log(myList[i], myList[i + 1]);
  }
}

// hacky way
var myList = [1, 2, 3, 4, 5, 6, 7, 8];

// we just append the first element to the end
// myList is now [1,2,3,4,5,6,7,8,1]
myList = myList.concat(myList[0]);
var indexOfLast = 8;

for (var i = 0; i < 8; i++) {
  console.log(myList[i], myList[i + 1]);
}

It’s very similar to “Array Rotation” (Courtesy:- Henre Botha)

3. Bank Withdrawal Checking

I read this in the book The Go Programming Language by Alan Donovan and Brian Kernighan.

This is not a hack nor does it reduce work. I felt it gives a different approach to the problem. Many might disagree.

var balance int
func Deposit(amount int) {
    balance = balance + amount
}

// normally we check if balance is greater
// than the amount to be withdrawn
func Withdraw(amount int) bool {
    if balance >= amount {
        balance = balance - amount
        return true // withdrawal was a success
    }

    return false // failed
}

// A different way
func Withdraw(amount int) bool {
    Deposit(-amount) // we use deposit to make a negative deposit
    if balance < 0 {
        // if balance is less than 0, we add the original amount back
        Deposit(amount)
        return false
    }

    return true
}

4. Rounding to the Nearest Integer

I don’t recall the source.

**Problem:-**Rounding floats to their nearest complete values.
5.1 -> 5.0
5.7 -> 6.0

# Normal way
round(5.1, 0) # returns 5.0
round(5.7, 0) # returns 6.0

# A different way
import math

def my_round(float_please):
    # Please google math.ceil if you arent' aware of what it does
    # Just think about the different scenarios. I'm
    # pretty sure you won't need an explanation :)
    return math.ceil(float_please - 0.5)

my_round(5.1) # returns 5.0
my_round(5.7) # return 6.0

End

I’ll try my best to keep note of the link or repository that house such solutions from this point onwards. Also if you are aware of similar hacks or alternate ways of solving problems, please comment them below :)

Raft (A Consensus Algorithm) Explained

Sat, 23 Sep 2017 00:00:00 GMT

An attempt to understand and annotate Raft. I’ll also be linking to other resources that helped me get a better grip of the material in this paper. I will not contrast Raft and Paxos as I do not understand Paxos. Also I’ve left out 3 small topics (Cluster Changes, Log Compaction and Client Interaction) that are quite easy to grok once you understand the crux of the algorithm.

Paper:- Raft
Recommended By:- Shrayas Rajagopal

Algorithm Explained

You can kind of see Raft in action here - Raft Visually Explained. I’m pretty sure I won’t be able to explain it better than them. You can come back to this article for some of the smaller not so important details that isn’t covered by this visualisation.

Basics

To increase understandability they focused on two general approaches of problem simplification. The consensus problem is broken into three relatively independent subproblems (Leader Election, Log or State Replication, Safety). They’ve tried reducing the state space whenever possible by using randomised approaches
They’ve emphasised on the importance of having a leader. It greatly simplifies decision making
Leaders typically operate until they fail
At any given time each server is a leader, follower or a candidate. When a node first comes to life it always is a follower. If it doesn’t receive a heartbeat(an acknowledgement from the Leader) message for a specified time, it assumes there is no leader and gets promoted to candidate. It nominates itself and starts an election. If that particular node wins majority it becomes the leader for that term
Each server can vote for one candidate on a first-come-first-serve basis. Assume node 1 and 2 have timed out and are requesting for votes. Node 1 reaches out to node 3 before node 2, node 3 will vote in favour of node 1
Time is divided into units called terms. Terms are represented by positive consecutive integers. Each node keeps track of
Terms act as a logical clock. Say a node becomes the leader for term 2 and becomes unavailable (network cable unplugged or whatever) after some time. One of the functional follower nodes timeout and enter candidate state and eventually becomes the leader. When the unavailable leader return to the cluster it notices that it’s term 3 (remember it was the leader for term 2), it steps down and becomes a follower
Terms are also used to find and fix anomalies in data between different nodes
Servers communicate using Remote Procedure Calls. They are issued in parallel for performance gains
There can only be one leader for a particular term. Say servers become lose connectivity and are split into two groups that can communicate amongst themselves. One of the groups is now void of a leader. One of the nodes in the group without a leader initiates an election by becoming a candidate. It’s then chosen as a leader and all the nodes of that group increment their term by one. Two different leaders can exist for these groups given that they are leaders for different terms

Leader Election

The entries (or change in state) flow through the leader to the other nodes in the system. Greatly reduces complexity and increased understandability
As explained earlier a candidate initiates an election if it doesn’t get a heartbeat for a said duration. One of the following can happen when an election is initiated
- It wins the vote if it secures a majority. Once it wins, it sends out heartbeats to other servers to claim authority and also to prevent elections unnecessarily. There are additionally safety checks before a candidate can take charge, I’ll cover them in a bit
- It receives a heartbeat or appendEntry (RPC to change state) from a leader during the voting process. It checks the term. If the term matches it’s current term or is higher, it realises that a new leader isn’t needed and transitions to become a follower. If the term is dated, it rejects the RPC as it probably is from the leader of an older term
- There’s a split vote or tie between two or more servers. Winning isn’t enough to become a leader, you need majority support. All the servers increment their term (this term had no leader) and initiate a re-election
Randomised timeouts are used to prevent split votes. If they had the same timeout, if a leader fails, many of the nodes might timeout in quick succession and the distributed system will witness many candidates. However if they had different timeouts, one of the node times out and becomes the leader even before other nodes get a chance to timeout

Log Replication

Once the troop has a leader, it begins responding to clients. Each client request contains a command to be executed by the replicated state machines (modify an existing entry or add new entries etc)
The leader makes the changes (or appends the client command to it’s log) and sends out RPCs (AppendEntries) in parallel to all the other nodes ordering them to do the same. Please note that the leader doesn’t commit the the entry just yet. The log entry is only committed when it is able to replicate the said change by the client in more than half the number of servers. It then returns the result of that execution to the client
A single or a few slow nodes will not impact performance at all. Once majority is achieved the entry is committed, the leader indefinitely tries sending RPCs to slow nodes until it gets a response
Log Matching Property (every command or entry has an index and term)
- If two entries have the same index and term, they store the same command
- If two entries in different logs have the same index and term, then the logs are identical in all preceding entries
The first property is a result of the fact that the leader creates at most one entry with a said index in a term. Also log entries never change position
The second property is maintained quite easily as well. When the leader sends out an AppendEntries RPC, it also includes the index and term of the preceding entry in it’s own log. If the follower’s latest entry does not match the preceding entry given in the RPC, it rejects it. It refuses new entries. This is a safety check mechanism. So if a follower responds to an AppendEntry the leader knows for sure that the particular follower’s log matches it’s log until that entry
If the follower rejects it’s AppendEntries RPC (implying inconsistencies), the leader looks up the logs of the follower. It finds the latest entry upto which they agree upon and delete every entry after that. It then sends all it’s entries to the follower after that point in time (they’ve also pointed a possible optimisation. The follower can send the conflicting entry instead of the leader having to search for it)
A leader never overwrites or deletes entries in it’s own log

Safety

Election Restriction
- The leader must have all committed entries
- When a candidate sends out a RequestVoteRPC, it also sends out details about it’s logs (last committed entry). The voter can deny to vote for that candidate if it’s logs are more up to date than the candidate
- Raft uses terms and entry index to determine the more recent log. If between two logs, one of them ends with a term 1 and the other has term 2, obviously the one with term 2 is more recent. If the logs have the same term, the longer log (therefore greater index) is more recent
Committing Entries From Previous Terms
- Say a leader copies entries to a majority of the servers but crashes before committing the entry
- A new leader is chosen. It is now implied that at least one of the server with the new uncommitted entry has voted for the new leader (because a new entry is accepted only with majority and the same goes for a new leader)
- It is important to keep in mind that only entries from the current term are committed by counting replicas across nodes
- In short previous term’s uncommitted entry if accepted by a majority of the nodes are automatically handled and committed when a new entry in a new term is received. They’ve explained the argument on page 9 of the paper

Miscellaneous

Follower and Candidate Crashes
They are quite easily handled. Say a follower crashes, the leader indefinitely tries sending it the same RPC. When the follower is back up, it then receives it and responds thus completing the cycle. Keep in mind that this wont’ effect performance as the leader only waits for the majority to respond. So a single or a few nodes don’t matter
Say a follower receives the RPC, appends the entries but fails before responding - When the follower is backup, it will receive the same RPC from the leader. Raft RPCs are idempotent. It notices the entry in it’s log and rejects it
Timing and Availability
This section explains something critical but very obvious for the proper functioning of the system. Say it takes 10 milliseconds to broadcast a message from the leader to all the followers the election timeout as you guessed it must be larger than 10 milliseconds (if it’s less than the broadcastTime, servers will keep timing out on a continuous basis assuming the absence of a leader)
MTBF is the average time between failures for a single server. It’s typically several months or more
MTBF and broadcastTime are properties of the underlying system. electionTimeout can however be manually set In short this should be broadcastTime < electionTimeout < MTBF MTBF should be much larger than the electionTimeout. When a leader crashes, the system will be unavailable for a duration roughly equivalent to the electionTimeout. Therefore the electionTimeout should be a very small portion of overall time. Say the leader crashes every 6 months or so and the electionTimeout is 500 milliseconds. The system is therefore unavailable for 500 ms every 6 months.

Python Threading With Simple Examples

Sun, 11 Jun 2017 00:00:00 GMT

I generally love the synchronous and predictable nature of Python. No promises, callbacks and nested code. Life’s good. Elegant and no time spent fixing async bugs. I rarely ever miss JavaScript’s async nature. Moreover I’ve hardly needed to use threads in Python.

However when using Python for network or disk intensive tasks, you start feeling the pain. JavaScript’s async nature would obviously make it much faster and you can imagine the time you’d save if you’d have say scrape large amounts of data and write them to disk on a regular basis. Also at times you might need threads to perform complex calculations faster.

Do keep in mind though - Python is limited by a Global Interpreter Lock (GIL). Threads you create don’t really execute code in a parallel fashion.

Creating Threads

Having additional threads can significantly increase the complexity of your program. You will need to ensure that they don’t compete for the same resources at the same time (race conditions), Global Interpreter Locks in Python, ensure that certain statements execute only after all the threads are done executing their assigned tasks and so on.

import threading
import requests

def fetch(url):
    response = requests.get(url)
    print "Got the url you asked for\n Status Code:-"
    print response.status_code

thread = threading.Thread(target=fetch, args=('http://www.example.com',))
thread.start()
print "Exit"

In the example above print "Exit" is called as soon the thread is created and will appear on the console before the print "Got the url you asked for. We shall see how to overcome this in the join section.

Now let’s try creating an arbitrary number of threads and also store them so that we can do other stuff with them later on.

import threading
import requests

def fetch(url, thread_number):
    response = requests.get(url)
    print "\n".join(["Hi, I'm thread number " + str(thread_number), "Here's the status code " + str(response.status_code)])

thread_list = []

for i in xrange(0, 5):
    thread_list.append(threading.Thread(target=fetch, args=('http://www.example.com', i,)))
    thread_list[i].start()

Notice how we store the threads in a list. This time I’m also supplying an integer so that we can identify them. Hopefully it’ll make things easier later on.

Joins

You also need a mechanism to control the flow of code. Say you want to read the file only after a thread has made a network request and written something to file. Jump back to the first example on the page. print "Exit"

Once a join statement is called in context to a thread, execution pauses unless the thread is done with it’s task. Continuing example one.

import threading
import requests

def fetch(url):
    response = requests.get(url)
    print "Got the url you asked for\n Status Code:-"
    print response.status_code

thread = threading.Thread(target=fetch, args=('http://www.example.com',))
thread.start()
thread.join() # Code after this gets executed only after the thread is done executing.
print "Exit"

Comparing total time taken

Without using threads.
Time Taken - 25 seconds

import threading
import requests
import time

tic = time.time()

def fetch(url, thread_number):
    return requests.get(url)


for i in xrange(0, 25):
    print str(i) + " " + str(fetch('http://www.example.com', i).status_code)

toc = time.time()

print "Time Taken:-  " + str(round(toc - tic, 2)) + " seconds"

Using threads. I’ve blindly used 25 threads. In practice the number of threads are only set after careful analysis and trial and error.

Time Taken - 6 seconds

import threading
import requests
import time

tic = time.time()

def fetch(url, thread_number):
    status_code = requests.get(url).status_code
    print str(thread_number) + " " + str(status_code)

thread_list = []
for i in xrange(0, 25):
    thread_list.append(threading.Thread(target=fetch, args=('http://www.example.com', i,)))
    thread_list[i].start()

for i in xrange(0, 25):
    thread_list[i].join()

toc = time.time()

print "Time Taken:-  " + str(round(toc - tic, 2)) + " seconds"

In the example above I use multiple join statements in a loop. Let me brief you about the flow.

.join() is called on the first thread. So the main thread is blocked until thread one finishes execution.
Now the interpreter tries to .join() the second thread. Let’s assume it has already finished execution. It does not block the main thread and immediately calls .join() on the third thread.
This continues till the 25th thread.

Try Making Them Functional

You can reduce complexity when using threads if you reduce their impact on common state. Eg:- If you create threads that access the same file, you might run into race conditions. You would have to lock and unlock the resource and so on. You could perhaps structure them in such a way that they read and write to their own files.

If performing complex calculations using threads, organise them in such a way that they take in inputs and give out outputs. That’s it. By inputs and outputs I mean they shouldn’t change the value of a variable or write disk to file or do anything that might effect something else. Think about it. By having functions that don’t change anything externally, you greatly reduce the complexity of your code and you also make testing easier.

How Many Threads?

There’s no magic number. Also the advise to create threads equivalent to 2 * number of CPU cores is hardly applicable and you should refrain from following it. Also be aware of the Global Interpreter Lock. I’d suggest reading the following StackOverflow answers How many threads is too many? and The right way to limit maximum number of threads.

Downloading Stuff with Firefox and Selenium

Thu, 02 Feb 2017 00:00:00 GMT

Index

Selenium Not Required
Selenium Becomes Necessary

Selenium Not Required

Consider this small example where curl or Python’s urllib are sufficient. You want to download 1000 zip files from URI’s having a set pattern.

examplesite.com/download/1.zip
examplesite.com/download/2.zip
examplesite.com/download/3.zip and so on.

In this case you could write a Bash or a Python script that keeps incrementing an integer, append it to the desired string, fetch the content of the URI and save the content in your desired location. Emulating a browser doesn’t even make sense in this case. It’ll be much slower due to GUI coming into picture, might not work on different versions of Firefox (I’ve faced this issue. Selenium often stops working with new updated versions of Firefox. It’s a good idea to turn off auto update in case you use Selenium often).

Selenium Becomes Necessary

You’ve probably used Selenium to automate tests for your web application or to scrape some data from a few websites. It is also possible to use Selenium to download stuff automatically and save them in your desired location. Selenium might be handy in the following conditions.

The app uses JavaScript and HTML5 to create a CSV that can then be downloaded.
Download link is randomly generated by the server. It changes from time to time or there is no recognisable pattern in a bunch of files.
Some websites try blocking scripts (programmatic access) and only access via browsers (real humans). They have some mechanisms to help them achieve the same.

Pesky Confirmation Dialogs

You can press download buttons using Selenium. But what about those confirmation boxes that appear asking you if you really want to download that particular file? Selenium as of today has no way to interact with those confirmation dialog boxes.

Alternative One

Firefox has various options pertaining to dowloads like default location, alert on complete, auto download etc. You can make system wide changes on Firefox and you might have to fiddle with some advanced settings on Firefox (visit about:config on your browser to change such settings). So you can tune it such that it doesn’t ask for confirmation nor the location thus avoiding those dialog boxes. The problem with this approach is that your code isn’t going to work on your friend’s system out of the box and these changes are going to effect your browsing experience.

Enter Firefox Profiles

Firefox provides an API to change it’s configuration. We’ll be using Selenium to interact with the API and temporarily use a particular configuration for our purposes. This increases the probability of it working on different computers and also gives you a way to preserve your default settings.

After a decent amount of time spent on StackOverflow and Google and testing various methods given by people I found the following code to work for me :)

from selenium import webdriver


fp = webdriver.FirefoxProfile()
fp.set_preference("browser.download.folderList", 2)
fp.set_preference("browser.download.manager.showWhenStarting", False)
fp.set_preference("browser.download.dir", os.path.join(os.getcwd(), data_directory_path))
fp.set_preference("browser.helperApps.neverAsk.saveToDisk", "application/csv, text/csv, application/pdfss, text/csv, application/excel")
fp.set_preference("browser.download.manager.showAlertOnComplete", False)

driver = webdriver.Firefox(firefox_profile=fp)
# Followed by code to download the stuff you desire

CS Journey's RSS Feed

Life's Great. Things Could Always Get a Whole Lot Worse - Stories from 2025.

Song Analysis - Fix You by Coldplay

General

Verse 1 and 2

Chorus 1

Song Analysis - Baarishein by Anuv Jain

General

Verse 1

Chorus 1

Bridge

Outro

Song Analysis - Perfect by Ed Sheeran

General

Verse

Pre-Chorus

Chorus

(WIP) Python cachetools - Source Code Walk

Constraints are Real. For some Acceptance Comes after Hesitation.

Grandmother, Music and Memory

Portfolio - Live Algo Trading Platform

Index

Context

Issues with Paid APIs

Objectives

Features

Distributed Web Sockets

Run Everywhere, Control from Anywhere

Centralised Risk Management

Unified Interface

Ease of Access

Tech Stack

Learning to Think in React by Building a Shopping Cart

30 Days Road Map - Budding Front End Developer to your First Job

Index

What do they Look for?

About Me

Bigger Picture

Build, Build and Build

Decent Portfolio

Plan

Day 1 - 7

Prerequisites

Goals

Plan

Day 8 - 14

Prerequisites

Goals

Plan

Day 15 - 21

Prerequisites

Goals

Plan

Day 22 - 28

Prerequisites

Goals

Plan

Day 29 & 30

Plan

Beyond Day 30

Advanced List

Credits

Managing Processes with Supervisor - In Depth Tutorial

Index

Basics

Installing Supervisor

Writing conf Files

A Few Options Explained

Directory

Command

Stop and Kill as Group

Logs

Some Common Errors

Vuvuzela - A Private Messaging System

Recommended Reading

Metadata

Existing Work

Todo

Interesting Stuff Mentioned in the Meetup

Learning to Think in React by Building a Stopwatch