Puppet's optional parameters might not be what you think they are

Photo by Kevin Ku: https://www.pexels.com/photo/data-codes-through-eyeglasses-577585/

Recently I encountered a puppet-lint warning regarding one of the classes I maintain at a customer. And this warning lead me down a rabbit hole of things I knew, things I understood and things I didn't know and apparently haven't grasp fully before. Surprisingly I never encountered this warning before. Despite being, most likely, a pretty common one.

So, I learned a bit or two about "optional parameters" in Puppet and thought this will give a decent read for my blog.

The problem

Take the following class definition. The code/logic is omitted as it doesn't matter (likewise I removed the puppet-strings documentation at the start of the manifests). We can purely focus on the class definition.

class profile::services::test (
  String $blatest,
  String $password = "test",
  Pattern[/^\d+\.\d+\.\d+(?:-\d+)?$/] $version = '11.2.3-4',
  String $system = 'SOMEEXAMPELSTRING',
  Optional[Hash[Stdlib::Port, Stdlib::Host]] $backupservers,
) {
# intentionally left empty
}

puppet-lint printed out the following warning:

WARNING: optional parameter listed before required parameter on line 6 (check: parameter_order)

And I was stumped by it. "Line 6 IS an optional parameter!" I thought. "There is no parameter defined after this, so how can it be listed before a required parameter?" I was confused. I moved line 6 around and sometimes the warning would vanished, sometimes not. Always defying what I thought I knew was correct.

The documentation is always right?

When you pass a certain amount of time you should stop to fiddle around and start solving it in an iterative way. So I went on to read the documentation. Thinking that maybe I missed some change in the Puppet DSL.

That's when I came across the paragraph on "Display order of parameters" as it mentioned the term "optional parameter". However.. It defined it in a totally different way I was used to use this term.

Display order of parameters

In parameterized class and defined resource type definitions, you can list required parameters before optional parameters (that is, parameters with defaults). Required parameters are parameters that are not set to anything, including undef. For example, parameters such as passwords or IP addresses might not have reasonable default values.

You can also group related parameters, order them alphabetically, or in the order you encounter them in the code. How you order parameters is personal preference.

Note that treating a parameter like a namevar and defaulting it to $title or $name does not make it a required parameter. It should still be listed following the order recommended here.

Source: The Puppet language style guide: Display order of parameters

And here I learned that: Any parameter with an associated default-value is called an optional parameter. But what about the Optional-data type?

Puppet goes on to give us one good and one bad example regarding the ordering:

Good:

class dhcp (
  $dnsdomain,
  $nameservers,
  $default_lease_time = 3600,
  $max_lease_time     = 86400,
) {}

Bad:

class ntp (
  $options   = "iburst",
  $servers,
  $multicast = false,
) {}

This recommendation is the reason why the puppetlabs-puppet-lint Gem enforces the order of: Required parameters first, optional parameters last. Where "optional parameter" again means: Any parameter with an associated default-value.

I changed line 6 by setting an empty hash as the default-value and the puppet-lint warning was gone. I also could have moved line 6 above or below line 2 (String $blatest,). Which would have fixed the warning too.

But here it is more or less the coding-style to list Optional-data type parameters after all others. Regardless of being a required or optional parameter. And again, this is being done here this way as people watch them as "optional parameters" in the sense of "I don't have to use if I don't need them". And here the requirement for parameter ordering bites the official definition of "optional parameters"...

The solution:

class profile::services::test (
  String $blatest,
  String $password = "test",
  Pattern[/^\d+\.\d+\.\d+(?:-\d+)?$/] $version = '11.2.3-4',
  String $system = 'SOMEEXAMPELSTRING',
  Optional[Hash[Stdlib::Port, Stdlib::Host]] $backupservers = {},
) {
# intentionally left empty
}

A bit more explanation

Now that we have clarified what an optional parameter is: You are either already confused, or ask yourself where the confusion originates from. So, without further ado, let me introduce you to the Optional data type.

The Optional data type

The Optional data type wraps one other data type, and results in a data type that matches anything that type would match plus undef. This is useful for matching values that are allowed to be absent. It takes one required parameter.

Source: Puppert: Values, data types, and aliases: The Optional data type

Most likely you have seen that parameters in Puppet classes are defined with an identifier of the type of data they will store. Same as it's required in any programming language where you have to define type of the variable (Integer, Float, Boolean, String, etc.). The only difference to Puppet is that in Puppet this is not enforced (as to avoid the term "optional" here. 😅)

If we adopt the given "good example" from above, we can make it look like this:

class dhcp (
  String $dnsdomain,
  Hash[String] $nameservers,
  Optional[Integer] $default_lease_time = 3600,
  Integer $max_lease_time               = 86400,
) {}

Here $default_lease_time is an optional parameter, right? Yes, but not because it uses the Optional data type. It's only an optional parameter because it has a default-value of 3600 assigned. Likewise $max_lease_time is an optional parameter too.

And here the problem originates.

Most of people I encountered use the term "optional parameter" to refer to the $default_lease_time parameter and not $max_lease_time. Which is absolutely understandable given how your speech works, how we like to name things and how we combine those two.

Adding to this is the fact that parameters which are initialized with the additional Optional-data type are allowed to be undef additionally to the other types of data defined. Effectively making it optional to assign a value to them. And this is exactly how these type of parameters are often used. Hence the term "optional parameter" sounds logical when referring to these parameters.

Technically however, that's wrong. Alas in my case it was the reason why I spent more time than I liked fixing that one linter warning. But I learned something along that way. I think that's what matters, right?

A little bit more confusion?

When reading through all data language types definitions and data types I found a data type I knowingly never encountered before. The Undef and NotUndef data types.

The Undef data type

The data type of undef is Undef. It matches only the value undef, and takes no parameters.

Several abstract data types can match the undef value:

• The Data type matches undef in addition to several other data types.
• The Any type matches any value, including undef.
• The Optional type wraps one other data type, and returns a type that matches undef in addition to that type.
• The Variant type can accept the Undef type as a parameter, which makes the resulting data type match undef.
• The NotUndef type matches any value except undef.

Source: Puppert: Values, data types, and aliases: The Undef data type

The NotUndef data type

The NotUndef type matches any value except undef. It can also wrap one other data type, resulting in a type that matches anything the original type would match except undef. It accepts one optional parameter.

Source: Puppert: Values, data types, and aliases: The NotUndef data type

And yes, they are useful for what they are designed. But I REALLY wished Puppet would rename that Optional data-type to something else. (Is: CanBeUndef still available? 😂)

Conclusion

All definition problems aside: This shows again why the usage of proper development/analyze tools like Linters is so important. It not only helps reducing errors, no. It forces you to understand what you are doing. 😉