As it turns out, generating pdfs at scale and in a cost effective way is non-trivial. In fact, it sucks.
This article is part of a series that outlines all of the problems I have faced with pdfs. If you want to go to the solution right away, skip to the bottom.
Part I: Just Getting it Working
It starts off with a simple request. My client wants to print a report to pdf. Easy enough. I would solve this by generating a report in html format and converting that html to a pdf. After a few google searches, it is quite clear that wkhtmltopdf and headless chrome are the front runners. I started off with wkhtmltopdf, an open-source solution for converting html and urls to pdfs.
I install wkhtmltopdf to my development machine (I already suspect this is going to be a problem since my client’s code runs on Heroku). After going through the install process, I now need to interact with it via code. The application I’m building itself is just a small Flask app written in Python.
I found a neat little wrapper for wkhtmltopdf in Python called pdfkit. I “pip install pdfkit” and get to work. With wkhtmltopdf being a .exe, I realize very quickly that I have another hurdle to overcome. I’m developing on a windows machine, but Heroku is a linux environment. In code, I need to detect which type of operating system I am running on so that it can always know how to load wkhtmltopdf. Here is my code for that. Double check to make sure your wkhtmltopdf binaries are in the right locations.
This does not solve my wkhtmltopdf on Heroku problem. After a bunch of searching, finding build-packs that do not work or are too hard to install / figure out, I found this github repo that has not been updated in several years. I got really lucky, and placed the github repo in my requirements.txt of my Flask app on Heroku: git+git://github.com/johnfraney/wkhtmltopdf-pack.git – and it automatically installs wkhtmltopdf binary in the right place. Unfortunately, I do not know if this repo will disappear or if it is no longer supported.
At this point, I successfully generate pdfs from html and urls with wkhtmltopdf and have it deployed on Heroku.
Part II: Scaling and Cost Explosion
Unfortunately for me, my customer is not just printing the occasional invoice or receipt to pdf. wkhtmltopdf eats up quite a bit of memory just sitting idle, but these are long reports with images and graphs. It is consuming all of our processing power and gigabytes of memory. Many, many reports are run in a short period of time too. The short term solution is to just continue to scale up the Heroku app. We went from spending $7/mo to $300/mo and there was no sign of slowing down.
The main problem is that we had no idea when or if a pdf would need to be generated that would consume quite a bit of memory. We also had no way of knowing if a lot of pdfs would be generated in a short period of time or spread out. As a result, this server had to be running at full-tilt at all times just in case.
Part III: More Requirements
While we are dealing with the price of our Heroku server going up and up – we were tasked with even more pdf challenges. No longer just convert html and urls to pdf, but to merge two or more existing pdfs into a single pdf.
Also, convert Microsoft office documents, like MS Word to Pdf and MS Excel to pdf.
After quite a bit of research and engineering we got it working by wrangling some tools and LibreOffice. But Heroku was long gone and we moved to a virtual machine just for our pdf generation.
With the increased processing power and memory needed for this pdf server to meet the new requirements, the server was costing over $1000/mo.
Our customers were literally generating pdfs that were gigabytes in size.
Time was running out. The server was crashing repeatedly and the clients were flipping out. We needed to do something completely different or the costs would continue to skyrocket. As it turns out, generating pdfs at scale and in a cost effective way is non-trivial. In fact, it sucks.
Part IV: Lambda
AWS Lambda is a wonderful service. It allows you to pay only for the compute cycles that you consume. I knew that if we could somehow utilize AWS Lambda, we could pay per pdf, instead of having to pay for a high powered server running at all hours. Additionally, lambda is infinitely scalable so we would not need to worry about the number of requests hitting the server at any given time. The problem is, trying to figure out how to get everything working in this cloud architecture is so frustrating. Nothing ever works as you expect it to. It took a couple weeks of development time to get all of our systems into AWS lambda and working well.
Incredibly, costs are down to about $50/mo and no more crashing server. Truly amazing difference.
If you want to skip all of the headache we had to go through detailed above, it is time to plug the service I launched which wraps wkhtmltopdf, headless chrome, merge / pdf concatentation, and libreoffice called Api2Pdf. Api2Pdf.com leverages the engineering we had to go through to make a completely scalable and cost effective solution for generating pdfs available to the public. It only costs $1/mo + $0.001 per megabyte. This is more than 90% cheaper than almost any existing service out there. The API is super easy to use and most customers will only end up spending about $2/mo. There are no monthly commitments and you can recharge your balance at any time.
The best part is you can get started with generating pdfs in minutes. I hope you like it and find it useful. Feel free to reach out to me on social media if you have any comments or feedback.