How to code sign Windows installers with an EV cert on GitHub Actions

This post walks through what it takes to code sign Windows installers in the cloud with an Extended Validation (EV) cert.

This process is surprisingly difficult and not well documented. It’s enough of a problem that I cold-emailed the CEO of GitHub about the difficulty of building deployable Windows apps on GitHub infrastructure and ended up speaking with several helpful members of the GitHub Actions team about improving the process and documentation.

Until then, come navigate this disastrous-and-just-barely-functional world of enterprise code signing with me.

Why Bother?

Your installer will throw up a Malware warning on Windows by default.

That’s a bad thing. Installation is the start of the user’s first experience with your product. Adding friction there is unfriendly — especially for less technical users, who might not know what action to take. (Microsoft adding this much friction to the code signing process is also unfriendly, but hey!)

To resolve this friction, you’ll need to cough up and pay the certificate mafia a yearly fee for an “Extended Validation” (EV) code signing certificate.

No, you don’t want the (cheaper) OV cert. That’s a pain in the neck and will still throw up Malware warnings under circumstances you can’t control — especially while getting started as a business. Only the EV cert solves the issue completely.

What’s the damage?

On macOS, you pay a flat rate to Apple to have a dev account — a code signing cert and service is just one of the many benefits. Plus, you can store that cert itself as a secret in CI. Life is easy.

On Windows, it’s nasty:

  1. You need a EV cert from a third party issuer (not Microsoft). You’ll have the absolute pleasure of selecting which one of the cabal of slow moving, expensive enterprise companies (sporting websites looking like they were made in 1999) you’d like your soul crushed by.
  2. It’ll cost you to interact with these cert issuers, both in time and money. I paid 709 EUR for a 3 year cert.
  3. The “old school” method is still dominant: Certs are deployed on a physical dongle, referred to as a Hardware Security Module (HSM). That’s pretty incompatible with modern development workflows that use CI… which is the motivation for this blog post.
  4. Luckily, the industry is in transition (read: 🐢🐢🤑) and both Google and Azure offer “cloud HSM” services. So…. Microsoft won’t offer you the cert, but they’ll host it. And you’ll need to pay a bit ($5/mo) for opportunity to store your cert on a cloud HSM.

We’re going to use Azure to store the key and GitHub Actions to do the signing. If you just want to fast forward to the end result, check out my Pamplejuce template on GitHub.

Prefer Google as your corporate overlord of choice? Apparently someone made a GitHub Action for Google KMS and Michał has written an extremely extensive article on his journey with Google KMS. 

Prefer Amazon? Sorry, they don’t support OV/EV storage at this time.

Step 1: Get the EV cert process started

I went with GlobalSign (no particular endorsement) because they advertise and have explicit support for using Azure as certificate storage.

It cost me 709 EUR for a 3 year cert. I’m hoping to set-it-and-forget-it for 3 years, because despite how much fun I’m having, I don’t want this to be an annual event.

The vetting process takes a while (weeks). It can take longer (months) if you reside in a smaller country and have documents (business license, proof of address, ID) that require notarized translations. My German documents were accepted as-is, and obviously English documents are fine, too.

Regardless, place your order and get started early. They might want things from you like a letter from your accountant, snail mail to confirm your address, etc.

Step 2: Sign up with Azure & create an Azure Key Vault

Azure offers a “Premium” HSM product as a part of Azure Key Vault.

I haven’t used Azure before this. It looks a lot like Amazon AWS — wayyy too many services with wayyyy too much config and wayyyyy too much admin. (People now have full time jobs specializing and getting certified in these platforms!)

After you sign up, go ahead and create a Key Vault.

Again, there are two flavors of Vaults. We’ll need the “Premium” HSM Key Vault which lets you store the fancy HSM certs.

It’s currently $5/mo. They sometimes have promotions going on that offer free service for a year, etc.

A reader was confused by the fact that Azure also offers dedicated HSM. This is for an expensive, single tenant, full-control-over-the-hardware type thing. You don’t need this. You want the Key Vault shared HSM.

Note your key vault name, the subscription-id, the resource-group and the uhhh… the Key Vault’s https URI (🤷‍♂️ don’t ask). We’ll need that all later.

Step 3: Create a CSR and generate your cert

Once you are vetted and able to generate a certificate by your favorite Third Party Corporate Vulture:

1. Create a Certificate Signing Request (CSR) (on Azure)
2. Use the CSR to generate the actual code signing cert (with your third party)
3. Merge the certs (on Azure)

Here’s GlobalSign’s (of course slightly out of date) instructions.

Important: In the “Advanced Policy Configuration,” make sure “Exportable Private Key” is set to “No”, which will then let you select RSA-HSM as the key type.

RSA-HSM key type is required by GlobalSign as of June 2023. Choosing this means Azure will store the certificate on FIPS 140-2 level 2 certified HSM.

Thanks to Michał, who commented GlobalSign’s docs are out of date and don’t specify the requirement for the RSA-HSM key type as of June 1st, 2023.

This step is actually pretty quick, but given the amount of waiting that happens before you get your cert, it’s important to get it right!

Note the certificate name you used on Azure — you’ll need it for the signing step later.

Step 4: Setup an Azure “Application”

Install the Azure CLI locally. Assuming macOS, you can brew install azure-cli.

This will pop open a browser window to authenticate you:

az login

Next, ask your local wizard for the precise incantation and/or decipher this cryptic disaster. You’ll need the subscription-id (it’s a uuid) and resource-group (it’s a name) from Azure.

az ad sp create-for-rbac --name "myApp" --role contributor --scopes /subscriptions/{subscription-id}/resourceGroups/{resource-group} --sdk-auth

myApp is arbitrary. It’s the name of the Azure “Application” you are creating which will do the signing. You could call it “GitHub” or whatever you call your CI workflow.

You’ll get back something magnificent:

  {
    "clientId": "{uuid}",
    "clientSecret": "{uuid}",
    "subscriptionId": "{uuid}",
    "tenantId": "{uuid}",
  }

Make sure to save this entire awkward json result somewhere safe for now. You’ll need to reference it.

Step 5: Add permissions to your Azure client

Use the clientId uuid you just got and the key vault name to give your client the appropriate permissions. You can also do this through the UI if you like.

az keyvault set-policy -n {keyVaultName} --key-permissions sign --spn {clientId}

Disregard the garbage output. Toss it out with your indie integrity. Er, you can manually inspect the output json to check to be sure the actual permissions were added.

Oh, but don’t forget: We need to actually get the cert before we can sign with it:

keyvault set-policy -n {keyVaultName} --certificate-permissions get --spn {clientId}

You can verify all went well on the website.

Navigate to your Key Vault. Under “Settings” click “Access Policies”. You should see an “Application” with the following policies in place:

Sign and Get. Get and Sign.

Step 6: Package things up

Do you have something to actually sign yet?

InnoSetup is what many people package Windows installers with.

It is on the list of pre-installed software on GitHub Action’s windows-latest (Windows Server 2022) environment. So on GitHub you don’t need to install anything to create an installer that installs things.

Yes, msix is a thing. If you can figure out how to do that nicely with a few lines of easy to read config, let me know! It looked like an unfriendly birds nest of xml and json to me. 🪹🍝

Step 7: Sign with Azure SignTool

Things get a bit hairy now.

Azure provides docs around accessing keys from GitHub Actions, however they reference a now-deprecated Azure-official GitHub Action which leads to some dead end suggesting to “pass a custom script” to the more general Azure CLI Action.

Corporate dysfunction aside, our goal is to use something something like signtool (Window’s code signing program) to sign things with the cert.

Enter Azure SignTool.

This (somewhat confusingly) is an unofficial project run by a GitHub employee, but referenced by official Microsoft documentation as The Way to Sign with Azure Key Vault and The Way To Use Key Vault for GitHub Actions Secrets. However those are only pieces of the puzzle and contain some issues, so I recommend following this flow:

Add AzureSignTool as a step in CI like so:

dotnet tool install --global AzureSignTool 

Note that you don’t need to run the az tool to login, like you might see in other guides.

Ready for more magic incantations?🪄

You’ll run this command to actually sign your installer:

AzureSignTool sign -kvu "${{ secrets.AZURE_KEY_VAULT_URI }}" -kvi "${{ secrets.AZURE_CLIENT_ID }}" -kvt "${{ secrets.AZURE_TENANT_ID }}"-kvs "${{ secrets.AZURE_CLIENT_SECRET }}" -kvc ${{ secrets.AZURE_CERT_NAME }} -tr http://timestamp.digicert.com -v yourInstaller.exe

This means adding the following 5 secrets to your GitHub repo:

AZURE_KEY_VAULT_URI For unknown reasons, this is an https URI? It is located on your key vault page
AZURE_CLIENT_ID (uuid from the awkward json result we had above)
AZURE_CLIENT_SECRET(string from the awkward json)
AZURE_CERT_NAME This is just the name of your cert on Azure, you can grab it from the UI. Note that the official Microsoft docs for pipeline signing call this “AzureKeyVaultName” which is terrible — both the key vault and the certificate have names. You’ll want to pass the certificate name to -kvc.
AZURE_TENANT_ID (uuid from awkward json)

The 5 secrets to uhh… Azure-ing your success in CI

You did it!

This is what the final result might look like on GitHub Actions:

    - name: Generate Installer and Sign with EV cert on Azure (Windows)
      if: ${{ matrix.name == 'Windows' }}
      shell: bash
      run: |
        iscc "packaging\installer.iss"
        mv "packaging/Output/${{ env.PRODUCT_NAME }}.exe" ${{ env.ARTIFACTS_PATH }}/
        dotnet tool install --global AzureSignTool 
        AzureSignTool sign -kvu "${{ secrets.AZURE_KEY_VAULT_URI }}" -kvi "${{ secrets.AZURE_CLIENT_ID }}" -kvt "${{ secrets.AZURE_TENANT_ID }}" -kvs "${{ secrets.AZURE_CLIENT_SECRET }}" -kvc ${{ secrets.AZURE_CERT_NAME }} -tr http://timestamp.digicert.com -v "${{ env.ARTIFACTS_PATH }}/${{ env.PRODUCT_NAME }}.exe"

You can check Pamplejuce, my C++/JUCE audio plugin template repo which runs this action on every build.

Troubleshooting: Azure Permissions

If you get far enough along that you are having permission issues — congratulations! So close!

Luckily the az error output is actually pretty decent! For example, I was missing the “certificate get permission” and got the error does not have certificates get permission on key vault.

Troubleshooting: A certificate with (name/id) *** was not found in this key vault.

I ran into this one. Make sure you are passing your certificate name, not your key vault name as the value to -kvc.


Responses

  1. Moritz Avatar
    Moritz

    Did somebody find a way to sign AAX with an cloud EV cert? Since most PACE stuff is under NDA, feel free to email me: moritz@noiseworksaudio.com

  2. Jie Avatar
    Jie

    Hi,

    I somehow followed the original instruction by GlobalSign and I set the type to RSA and the CSR was accepted by GlobalSign. Does it mean RSA should work fine too?

    1. Jie Avatar
      Jie

      I asked bing and it gives following answer

      Azure Key Vault is FIPS 140-2 Level 2 compliant and only supports asymmetric keys. It also supports RSA keys of sizes 2048, 3072 and 40961.

      So if you generate an RSA certificate from Azure Key Vault, it will be FIPS 140-2 Level 2 compliant

  3. Michał Leszczyński Avatar

    Just be aware, right now the CA might ask you to show the attestation proof that the code signing private key is stored in FIPS 140-2 Level 2 compliant device.

    Following the Microsoft Azure key generation guide which was mentioned in this article would result in generation of “Software-protected” RSA key. This is compliant with FIPS 140-2 Level 1(!) only – so the compliance requirements are not met here.

    Moreover, even if you would somehow do it differently and have the proper RSA-HSM key generated in the Key Vault, it’s still not possible to download the attestation proof as of now:
    https://learn.microsoft.com/en-us/answers/questions/1185506/current-azure-key-vault-fips-140-2-level-2-proof

    I’ve switched to Google Cloud KMS because it allows to generate the HSM proof, which seems to be mandatory to have now.

    1. sudara Avatar

      I agree the situation is confusing and information seems limited around this topic.

      However, GlobalSign proactively confirmed to me on the phone that nothing has changed with regards to their Azure support and that no action would need to be taken for use cases with Azure Key Vault. That leads me to believe that as an end user you do not need to provide attestation when using Azure (it must be allowlisted?).

      It’s always possible there’s some organizational dysfunction and there will be problems in the transition. However, given that EV code signing deployed to Azure is one of their main products, I’m also assuming they will update their docs if changes are eventually required.

      Also, the link you provided confirms that the shared Azure Key Vault is FIPS 140-2 Level 2 compliant and provides links to the public certificates, so everything seems on track there as well…

      They sent out another email notification about these changes. I’ll followup to see if they can make a more public statement or update some documentation re: Azure and these changes.

    2. Michał Leszczyński Avatar

      Thanks for your response!

      As of now, it is required to sign a document with such a statement:
      > “Subscriber certifies that all EV Code Signing key materials were generated on and are to be stored on the FIPS 140-2 Level2 compliant HSM.”
      > “Note: At any time during the application and life-cycle of the certificate subscriber must be able to, on request of GlobalSign, present proof the key pair associated with the certificate (request) is stored on a cryptographic device that meets the requirements of FIPS 140-2 Level 2 (or equivalent). Failure to provide such evidence might result in revocation of the certificate.”

      It’s not really clear whether the screenshot from Azure Key Vault panel would be considered as a valid proof or not. I guess it would be, but there is no clear guarantee on that. Something surely considered as a strong proof is the signed HSM key attestation, but unfortunately, as far as I know it’s not possible to generate such thing out of Azure Key Vault at the moment.

      Also, please note that “RSA” type keys are only Level 1 compliant (doesn’t meet the requirements) and “RSA-HSM” type keys are Level 2 compliant indeed (meet the requirements). Following the guide linked by your article: “Generating and Importing a Certificate into Microsoft Azure Key Vault” leads to creation of “RSA” type key, which is only Level 1 and thus certainly not compliant with the requirements.

      So I belive that the key should be created using different procedure, so to arrive at “RSA-HSM” (not “RSA”) key type. Then it will be technically compliant with FIPS 140-2 Level 2.

    3. sudara Avatar

      Ok, I finally see what you mean by the GlobalSign guide. They need to update their docs to make it clear that HSM has to be selected. I’ll update the article with an additional screenshot showing how to select HSM.

      In the article I do make clear that you need to use the “Premium” Key Vault which does support the RSA-HSM flavor, see:
      https://learn.microsoft.com/en-us/azure/key-vault/keys/about-keys

      Re: attestation, my operating assumption is GlobalSign has a relationship with Azure, who handles that. (Since you aren’t the entity hosting the cert, it doesn’t make sense to me that you’ll be asked to provide attestation, but I guess this stuff is messy!)

      Google Cloud KMS sounds like a good alternative though, let me know how you get along with signing there! Is there a good API/tool to do the job?

    4. Michał Leszczyński Avatar

      With Google Cloud KMS, it’s possible to perform all the initial setup inside the Google Cloud Console. The tricky part is to generate the CSR – you would need to install Google’s PKCS#11 library and configure openssl tool accordingly. The actual binary signing could be done using `jsign` Java tool which has the native support for Cloud KMS.

      I took an enormous amount of notes (and time) while figuring out how to do it all correctly, so I’ve synthetized it all to an article. It’s on my website under “Other articles” section.

      I also see that the article is updated already, cool, thanks for making it more precise around the RSA-HSM keys.

    5. Michał Leszczyński Avatar

      Also, thanks for writing this blog post as this was one of the initial knowledge sources for me setting this up. I’ve added a mention about that at the end of my article 🙂

    6. sudara Avatar

      Wow, your article is seriously detailed! Nice work!

      Good idea, I’ll also link your article at the start of mine when I mention Google KMS, as well as re: the RSA-HSM. There’s few resources on this, and as you have illustrated, first party information tends to be confusing as well.

      Thanks Michał!

    7. Michał Leszczyński Avatar

      Thanks for the cross-link. Wishing all the best with your projects!

  4. Roman Avatar
    Roman

    Hi Sudara, thanks a lot for sharing your experience! I am planning to install a new EV Code Signing certificate to Azure Key Vault, and I am also using GlobalSign, so your article is a great help.

    I have a question regarding the CSR generation. A guide from GlobalSign that you referred to does not mention which Extended Key Usages should be specified when creating a CSR on Azure. There is a screenshot though, which features the default value set by Azure: 1.3.6.1.5.5.7.3.1, 1.3.6.1.5.5.7.3.2. I found out these OIDs stand for “Server authentication” and “Client authentication” respectively — which is probably valid for SSL certificates, but not code signing.

    I found another guide in Microsoft blog: https://techcommunity.microsoft.com/t5/modern-work-app-consult-blog/signing-a-msix-package-with-azure-key-vault/ba-p/1436154
    Which uses the Code Signing EKU (1.3.6.1.5.5.7.3.3).

    My question is: have you added or change values in the EKU when creating your CSR? Does this even matter for our use case (code signing)?

    Thanks for help!

    1. sudara Avatar

      Hmm! It’s been a while, but I don’t remember specifically changing those EKUs. I’m trying to inspect the certificate now but not seeing where those show up. Perhaps they are related to the CSR itself, I’m not too familiar with Extended Key Usages, I just followed the docs from GlobalSign…

    2. Vincent Avatar
      Vincent

      GlobalSign will ignore any EKU or key usage you enter in the CSR and use the appropriate values when they issue the cert. They probably use a template.
      When I did mine, I left those values at their defaults in the Azure CSR form:

      EKUs:
      1.3.6.1.5.5.7.3.1, 1.3.6.1.5.5.7.3.2

      X.509 Key Usage Flags:
      Digital Signature
      Key Encipherment

      And when I got the cert, the values were:
      EKU:
      Code Signing (1.3.6.1.5.5.7.3.3)

      Key Usage:
      Digital Signature

  5. Christian van der Leeden Avatar
    Christian van der Leeden

    Worked perfectly! Thanks for sharing. For German GmbH the verification process lasted less than 1 week. Our Sectigo application for an OV is still in limbo somewhere, so Global Sign (at least with a German company) worked very well. Now I just have to convince electronforge to build and sign everything with azuresigntool and move it to github CI.

    1. sudara Avatar

      1 week isn’t bad!

  6. Mark Avatar
    Mark

    FYI – The dedicated Azure Dedicated HSM is $4.85 per hour not per month. That equates to $3,540 per month or about $42k per year. Changes coming in June 2023 that will require an Azure Dedicated HSM with an attestation that the cert is generated on compliant hardware and is non-exportable.

    1. sudara Avatar

      Hey Mark,

      You are looking at the “Dedicated” HSM pricing.

      The normal HSM pricing is here: https://azure.microsoft.com/en-us/pricing/details/key-vault/ and is currently $1 per key per month + $0.03/10,000 transactions.

      I’ll put a note somewhere clear so there’s less confusion!

  7. MGough Avatar

    Thanks for this guide Sudara! There’s a few commands you might want to doublecheck as at least one of them still references your keyvault name.

    I managed to get this working to sign an `.msi` file that is produced in my CI. Despite it being a convoluted/dated process I got the EV Certificate within a few days of placing the order!

    I also investigated other providers but the CloudBased HSM support from other providers really isn’t well documented, although one provider seems to offer it for at least $500 more…

    1. sudara Avatar

      Thanks for the heads up, took me a while to find the place you are referencing! Wow, a few days isn’t bad.

  8. Anna K Shipman Avatar

    I think you can get it at much cheaper price at https://signmycode.com/sectigo-ev-code-signing . This is also supports using Azure as Certificate Storage. (Disclaimer : I own this site!).

    1. sudara Avatar

      Thanks for sharing. $600/3 years is a bit nicer than the $700 at GlobalSign, but on the page you linked it says “Validation Type: OV” — Is that a typo? Users will still see the malware warning with an OV cert…

      Do you have docs anywhere for how to do the CSR dance with Azure or other providers?

Leave a Reply

Your email address will not be published. Required fields are marked *