Hussein Nasser

The elegant design of separating concerns in Apache Kafka creates tons of potential scalability improvements.

But also comes with tradeoffs.

Publisher, Consumer and a Broker

1 day ago | [YT] | 267

Hussein Nasser

If you have 1-3 years if experience in backend engineering you may enjoy my intermediate backend engineering course bundle

It consists of three fundamental courses in this order for an effective learning roadmap.

1- Fundamentals of Operating Systems
2- Fundamentals of Network Engineering
3- Fundamentals of Backend Engineering

The OS course polishes your knowledge about how the application work with the kernel, socket programming and file systems.

The network course introduces the bridge to the backend, how the application and the kernel sockets work the network.

Finally the backend course introduces you to the common backend design patterns how it interacts with both the OS and network to receive requests, process and write responses.

Head to courses.husseinnasser.com/bundle to learn more

1 week ago | [YT] | 295

Hussein Nasser

Memorizing trick programming questions and answering interviews flawlessly as you may know isn’t productive.

Coming up with your own solution on the fly, no matter how inefficient or bad you think it is 100 times better.

You will flawlessly one day build up to a better solution brick by brick.

For those are your bricks. Not somebody else’s.

1 week ago | [YT] | 488

Hussein Nasser

Postgres runs on port 5432, this is the responsibility of the Post master process.

However did you know that listening socket isn’t created until another process finish running first?

That is startup process (ST), responsible for WAL redo when the database crashes.

You cannot allow clients to connect to the database if the state isn’t consistent. However some databases do allow clients to connect (SQL Server comes to mind) they just block them from doing anything, it is a design choice.

Where the backend meet databases, polish your skills and learn more about the fundamentals at courses.husseinnasser.com I would recommend starting with the OS course followed by network, backend then databases.

Skip the Carousel

1 2

2 weeks ago | [YT] | 452

Hussein Nasser

If have 3-5 years of experience in the backend and want to elevate your skills, consider grabbing the advanced Backend engineering Bundle. It consists of three fundamental courses taken in this order for an effective learning roadmap.

1- Fundamentals of Backend Engineering
2- Troubleshooting Backend Systems
3- Real-Time Backend Systems with WebSockets

The backend course introduces you to the common backend design patterns and how it interacts with both the OS and network to receive requests, process and write responses.

In the Backend Troubleshooting course (and also my favorite) you get to find issues with slow performance, unexpected bugs, and identify exactly where the trouble with performance toolings.

And finally the real-time backend systems, my newest course, building backend applications is interesting, building a realtime one is twice is challenging and interesting.

Please note that the Fundamentals of Backend Engineering is an intermediate course that requires some prerequisites such as networking and operating systems fundamentals which I also offer as part of the intermediate backend engineering bundle. You can find this on my website.

Limited offer expired Nov 5th using These coupons, you can have all three courses $29.97. Grab them now here.

The Advanced Backend Engineering Bundle

Backend Course coupon: BACKEND-FALL25
www.udemy.com/course/fundamentals-of-backend-commu…

Troubleshooting Backend Course: TROUBLE-FALL25
www.udemy.com/course/discovering-backend-bottlenec…

Realtime Backends Course: REALTIME-FALL25
www.udemy.com/course/scalable-real-time-backends-w…

Find all courses here courses.husseinnasser.com/

3 weeks ago | [YT] | 276

Hussein Nasser

Probably nothing symbolizes tradeoffs than RAID in software engineering

In RAID1 we wanted redundancy in case of disk failure, so the OS (or the RAID driver) would mirror block of data to multiple drives. This gives redundancy but slows down writes (you are essentially writing the same data twice and bound by the slowest drive). Here you only get 50% of the total drive space.

In RAID0 we wanted performance, so we would stripe the block and “smear” it across multiple devices, this way we get faster writes (same block is split in half and each half written in parallel) but if any device fails we lose our data. Great for temp caches and data you don’t care to lose. You get access to the entire two drive space.

Subsequent RAID systems are built on top of those two balancing, Performance vs Reliability vs Budget.

You will notice software engineers make these trade off all the time in the entire stack.

Skip the Carousel

1 2

3 weeks ago | [YT] | 142

Hussein Nasser

Page tables provide the mapping between virtual memory and physical memory for each process. This means it needs to be as efficient and as fast as possible. I explore the inner workings of page tables in this episode.


Watch it on YouTube
https://youtu.be/aJpcysMJkvk

Skip the Carousel

1 2

3 weeks ago | [YT] | 19

Hussein Nasser

Free performance for full table scans in Postgres 17

You see, Postgres like most databases work with fixed size pages. Pretty much everything is in this format, indexes, table data, etc. Those pages are 8K in size, each page will have the rows, or index tuples and a fixed header. The pages are just bytes in files and they are read and cached in the buffer pool.

To read page 0, for example, you would call read on offset 0 for 8192 bytes, To read page 1 that is another read system call from offset 8193 for 8192, page 7 is offset 57,345 for 8192 and so on.

If table is 100 pages stored a file, to do a full table scan, we would be making 100 system calls, each system call had an overhead (I talk about all of that in my OS course).

The enhancement in Postgres 17 is to combine I/Os you can specify how much IO to combine, so technically while possible you can scan that entire table in one system call doesn’t mean its always a good idea of course and Ill talk about that.

This also seems to included a vectorized I/O, with preadv system call which takes an array of offsets and lengths for random reads.

The challenge will become how to not read too much, say I’m doing a seq scan to find something, I read page 0 and found it and quit I don’t need to read any more pages. With this feature I might read 10 pages in one I/O and pull all its content, put in shared buffers only to find my result in the first page (essentially wasting disk bandwidth, memory etc)

It is going to be interesting to balance this out.

Note that true postgres issues a kernel system call to read 8k at a time the kernel can be configured to read “ahead” little bit read more and cache it in the file system cache.

——-
Learn more about database and OS internals, check out my courses
Fundamentals of database engineering databases.win/
Fundamentals of operating systems oscourse.win/

4 weeks ago | [YT] | 238

Hussein Nasser

Let us indulge in some engineering beauties of MySQL. It rarely gets Love compared to Postgres.

Uber reduced their database locks by 94% when they upgraded to MySQL 8.0, thanks to this performance re-architecture.

We know that writing to the index may cause a structure change (btree rebalance) which can cause leaf pages, internal pages and the ROOT to split and update.

Allowing a read while the structure is being changed can cause corruptions, so we need to protect the structure and readers via physical locks or a mutexes (I talk about those in my OS course)

In version 5.6, MySQL InnoDB opted to do a global exclusive lock (X lock) on entire index when a rebalance is triggered, preventing reads from happening (reads take a shared S lock), even when the reads are going to a different part of the tree.

In 8.0 only the pages being restructured or written are X Locked (not the entire index) also a snapshot of those pages are saved so concurrent reads to those pages are allowed. This index is instead locked via a new intent SX Lock which allows shared S locks but prevents X Lock.

What better proof of this engineering marvel than Uber upgrading to MySQL 8 and reducing their database locks by 94% !

We know MySQL is struggling recently against Postgres and I covered that in another video but the engineerings marvels must be acknowledged regardless of the state of the product.

---
Watch my full deep dive coverage here

Advanced MySQL Index Locking Explained
https://youtu.be/MK24y7AmKTc

I also try to start crediting the Devs behind the work, this work is done by Zongzhi Chen

Read uber’s article www.uber.com/en-JO/blog/upgrading-ubers-mysql-flee…

1 month ago | [YT] | 319