At Rocket.net, our top priority has always been our customers’ success, especially during their most critical times when they need us most. Rather than following the rest of the hosting industry with some gimmicky Black Friday promotion (we’ve never run one of these), our entire team has focused on supporting our existing and new customers on their biggest traffic days of the year.

Sure, everyone brags about how great their support is, and believe me, there are many amazing people out there, and I give them massive credit. 

But at Rocket.net, we are different. We’re willing to go the distance for our customers and do whatever it takes to ensure they’re always online and as optimized as possible.

So, let’s dive into a real-world example of what happened today.

Situation Overview

Today, around 4:30 AM Eastern, one of our large Enterprise customers noticed a slowdown in checkouts and specific pages throughout their site. So, we rolled our sleeves up and got to work.

First, I’ll share what the customer’s setup is:

Enterprise 4
96 cores
256GB RAM
2TB NVMe Storage

NVMe storage plays a huge role here for high traffic due to its high read/write speeds during heavy workloads.

For example, this customer still sees around 700mb/s writes and over 6.2gb/s reads during peak hours; this is extremely important for PHP and MySQL.

Now, outside of the server specs, they’ve been pleased with the performance, but they started to notice a slowdown as things picked up. 

The server was running at 5% CPU usage with bursts to about 20-30%, so something did not seem right.

Identifying the Problem

To get started, I installed New Relic to start tracing. Typically, we use PHP X-Ray for this job, but in this case, New Relic was the tool used for the job due to its granularity.

Once I started receiving some data, the issue stuck out: The Action Scheduler!

Take a look at some of these traces. We’re normally not pleased with more than a second or two of execution time, but here, processes took tens of seconds and tens of minutes!

I knew there was a problem, and I assumed it had to do with task queueing, but I did not understand WHY this problem was occurring.

So, I started diving into the Action Scheduler documentation: https://actionscheduler.org/perf/

Similar to WP Cron, the action scheduler executes on POST requests to admin-ajax.php. If these requests take a long time to process, they hog resources, slowing down visitor experience and causing PHP threads to start queuing new requests, ultimately elevating response time across the board.

Solving the Problem

So, following the Action Scheduler documentation, I built two cron jobs to move the action scheduler to WP CLI, which uses resources outside of the web server so that it doesn’t compete with client requests. 

Job 1 (every 5 minutes):

/path/to/php /path/to/wp action-scheduler run –group=woocommerce-webhooks –batch-size=500 –path=/path/to/wp

Job 2 (every minute):

/path/to/php /path/to/wp action-scheduler run –exclude-groups=woocommerce-webhooks –path=/path/to/wp

Why two? Well, this site uses explicitly many webhooks to push data to its internal systems, which ultimately was the real culprit here. Running jobs via WP CLI for webhooks can take seconds or even 10+ minutes. 

For example, if this customer does a bulk edit on orders for 1000 orders, it will create 1000 scheduled tasks for webhooks at minimum. 

Knowing these webhooks can take up to 10 minutes to complete in the queue, we don’t want to create a bad customer experience by delaying emails, subscription renewals, etc, by 10+ minutes, so we have another job that runs just for all non-webhook tasks.

After creating the jobs, we followed Action Scheduler’s advice and had our customer install and enable the Action Scheduler—Disable Default Queue Runner plugin.

The Result

After moving the action scheduler to WP CLI, we saw incredible results on the site:

 
In the above screenshot, you can see where an admin-ajax.php request took 43 seconds! Then, you can see where they dropped to 625ms-1s after moving the action scheduler to WP CLI, an incredible difference in response time.

Now, this scenario does not apply to all sites. To put this into perspective, this site processes about 20,000 requests per minute and uses about 4.6TB of bandwidth a day… and it’s not even Black Friday or Cyber Monday yet!

Without this fix, I do not believe they would have survived Black Friday, Cyber Monday, or any other massive sale.

So, if you’re running a large WooCommerce site and are preparing for big sales like Black Friday, look into the action scheduler as you’re checking things out… or just give us a shout. 

We’d be happy to set you up and optimize your WooCommerce store in time for Black Friday and Cyber Monday (yes, we even work Thanksgiving).