DevOps

Using Pulumi to put your infrastructure into writing

Cloud resources grew in importance over the last years, and related technologies like container platforms are being adopted more and more. They are extremely useful for businesses especially when they have unpredictable workloads, but they can also be very complex. Keeping a good overview over a full infrastructure implementation can be quite difficult. Cloud infrastructures are complex, and tools like Pulumi can help you to manage them by putting them into writing and handling them like code.

Managing Infrastructure as code

IaC isn’t just a fancy new way of doing things – handling your infrastructure in the same way you handle code files comes with significant upsides. A lot of best practices dealing with code make working with these artifacts way less time consuming and failsafe:

  • IaC allows you to easily version your infrastructure. While modern cloud provider platforms like AWS provide ways to find historic data on configuration, it often means clicking through a multitude of interfaces or requiring advanced knowledge of the related CLI tools. For code however, state-of-the-art tools like BitBucket or GitHub allow you to easily look into and review changes between commits. This makes working in teams way easier and applying this to infrastructure configuration can save a lot of time.
  • Deploying applications takes a lot of time – servers starting up require a bit of patience, and with multiple interconnected applications in a system, this issue can quickly exacerbate. Continuous Development and Integration practices commonly adopted by businesses in the recent years automate this process as much as possible, freeing up valuable time. This also applies to IaC: tools like Terraform and Pulumi are built to be used in automation platforms like Jenkins, Bamboo and the like. Infrastructure is a central part of an application and automating it’s deployment allows you to easily test it together with your application as close to the production environment as possible.

A little more than a year ago, I published an article series on how to create a Kubernetes stack using AWS managed resources and Terraform. The technology has only grown more popular, but with rising interest in Infrastructure-as-Code in general, there are more and more platforms appearing on the market. One of them is Pulumi.

Pulumi – a modernized IaC alternative

Pulumi uses Node.js, C#, Python or GO to deploy to Azure, AWS and Google Cloud.

Pulumi goes with Node.js, C#, Python, GO, Azure, AWS and Google Cloud.

Like Terraform, Pulumi allows you to save your infrastructure as code. But the developers of Pulumi were able to use their experience with similar tools to build a different product, so there are some core differences:

  • Terraform uses HCL (HashiCorp Configuration Language). While there are some other uses for the language, it is not widely adopted and most developers will need to study a bit before setting up infrastructure scripts. Pulumi is built as a library for multiple languages that should be familiar to many developers – Node.js (Typescript and Javascript), Python, GO and C#. This comes with another advantage: while writing programmatic logic like loops and conditions is possible using HCL and Terraform, it is laborious and not what the language was made for. The programming languages Pulumi supports are just that, making cases like conditional infrastructure configurations much more pleasant to implement.
  • Infrastructure isn’t a one-and-done deal – it needs to keep running, and it often needs to change with time. IaC-platforms need to keep track of the current state of already deployed servers and resources so they can track and apply changes. Terraform does this with state management in files: when you set up Terraform, you configure it to save files locally or online, and these files are looked up every time Terraform evaluates changes to the infrastructure before sending the relevant requests to the server.
    Pulumi has a slightly different default approach. It uses a centralised dashboard to keep track of your deployed stacks, which you can find at app.pulumi.com. Your local Pulumi installation connects to a central server and saves the state of your infrastructure there, enabling this overview. Note that doesn’t mean you can’t manage your infrastructure state by yourself – Pulumi also supports customised state management, which makes sense especially when you consider security concerns – but the centralised approach makes experimenting with infrastructure and cloud providers just a little bit less involved.

While these key differences are important to consider, Pulumi is still built on Terraform. The providers (the packages that contain the code to connect to a cloud provider to build a resource) are derived from Terraform providers which are proven to work. Pulumi also integrates well with existing Terraform usage: it can for example consume state files created with Terraform. There is also a neat tool called ‚tf2pulumi‘ which can translate most Terraform scripts to code compatible with Pulumi. Transitioning from Terraform to Pulumi is therefore relatively easy and can be done in steps, which allows you for example to keep the foundational parts like AWS VPCs in Terraform while you transfer server infrastructure to Pulumi.

What does a Pulumi project look like?

Setting up a Pulumi project is simple. We use AWS as our cloud provider for our little example. First off, we need to configure the AWS side of things. You need to create a user to connect to your account, for example by using IAM in the AWS console. Save the access key and the secret key and set up an AWS profile to easily manage your credentials, or keep them around a bit and save them directly in the project in the next step. Then install Pulumi according to your system.

The Pulumi client allows you to easily set up a new project with a simple command: pulumi new aws-typescript creates files for the basic configuration in an interactive manner. Three files are created:

  • Pulumi.yaml contains the basic definitions for the project. Identifying information like the name and description can be found here. Pulumi uses this to properly load the project when you run it.

  • Pulumi.dev.yaml contains the further configuration for the project. You can reference the AWS profile you created earlier here so that Pulumi uses the correct user to connect to AWS. You can also directly save the access and secret keys in this file. Other provider specific configurations also go here.

    Note that you can use this configuration file to customise staging differences. You could for example create a

    Pulumi.test.yaml to set up a different configuration and then call

    pulumi up -c Pulumi.test.yaml to set up this alternative stage – and you can also use this in automated CI/CD to push changes to multiple environments.

  • index.ts  contains the actual resource we want to build. Pulumi generates some basic configuration for a S3 bucket here. You can add any resources here. Like Terraform, Pulumi allows you to organize your code files in any way you want – the overall sum of resources is evaluated when you execute Pulumi to apply the changes, so all Typescript files containing resources in this folder will be considered.

Once this is set up, you can easily deploy your stack or destroy your stack, using the CLI commands

pulumi up and 

pulumi destroy respectively. Note that you can keep track of your implemented stacks after you started them on app.pulumi.com.

With this set up, we can start to add resources piece by piece to build a more complex infrastructure. In another article, I will lead you through the process of setting up some resources in AWS and connect them together using Pulumi.