Sandi Verdev

Case study

Cutting a push-notification job's memory from 9 GB to 4 GB

FirstBeatMedia · Senior PHP Backend Engineer · Feb – Mar 2024

Problem

A cron job sends Firebase push notifications to the users of the mobile apps. Its memory grew with every user it processed and peaked at about 9 GB.

Sometimes it used all of the server's memory and the kernel's out-of-memory killer stopped it mid-run, so notifications were not sent.

What I did

  • Released each user's objects as soon as that user's notification was built, so memory no longer grew with the number of users.
  • Replaced lookups that loaded the full profile and permission objects with direct reads of the two values the job needs: profile completeness and membership level.
  • Accepted a small, documented duplication: the completeness calculation now exists in two places, with a note in each to keep them in sync, instead of loading the whole profile object for one number.
  • Refactored the publisher job first, so each step was small and easy to change.

Result

Peak RAM of the push jobBefore the change the job peaked at 9 GB of RAM; after, at 4 GB, 56% less.Peak RAM of the push-notification job, GB0369Before9 GBAfter4 GB · −56%
Peak RAM of the production job before and after the change. After the change the job was no longer killed mid-run.

Peak RAM is PHP's own peak-memory figure, logged for every run of the job and shown in the admin's cron monitoring.

Stack

  • PHP
  • MySQL
  • Firebase Cloud Messaging
  • cron