Composable roles tutorial

This guide will be a walkthrough related to how to add new services to a TripleO deployment through additions to the tripleo-heat-templates and puppet-tripleo repositories, using part of the architecture defined in the composable services architecture


The initial scope of this tutorial is to create a brief walkthrough with some guidelines and naming conventions for future modules and features aligned with the composable roles architecture. Regarding the described example in this tutorial, which leads to align a service implementation with the composable roles approach, is important to notice that a similar approach would be followed if a user needed to add an entirely new service to a tripleo deployment.

The first step will be to identify and isolate a functionality or feature that needs to be aligned with the composable roles approach. For instance, by checking tripleo-heat-templates/puppet/manifests/overcloud_controller.pp and selecting an installed service (i.e. ‘include ::ntp’).

The puppet manifests used to configure services on overcloud nodes currently reside in the tripleo-heat-templates repository, in puppet/manifest In order to properly organize and structure the code, all manifests will be re-defined in the puppet-tripleo repository, and adapted to the composable services architecture

The use case for this example uses NTP as a service installed by default among the OpenStack deployment. In this particular case, NTP is installed and configured based on the code from puppet manifests:

puppet/manifests/overcloud_cephstorage.pp, puppet/manifests/overcloud_volume.pp, puppet/manifests/overcloud_object.pp, puppet/manifests/overcloud_compute.pp, puppet/manifests/overcloud_controller.pp and puppet/manifests/overcloud_controller_pacemaker.pp

Which means that NTP will be installed everywhere in the overcloud, so there is no need to add more code. Instead we need to refactor the code from those files in order move it to puppet-tripleo.

Also it is important to notice that the NTP puppet module is already included in the repository tripleo-puppet-elements, so we can use the manifests from NTP in any node from the overcloud.

In the case of needing a new third-party puppet module, it should be added in tripleo-puppet-elements (installed by default in all nodes) and referenced from a puppet manifest in puppet-tripleo.

This tutorial is divided into several steps, according to different changes that need to be added to the structure of tripleo-heat-templates and puppet-tripleo.

Repositories referenced in this guide

  • tripleo-heat-templates: All the tripleo-heat-templates (aka THT) logic.
  • puppet-tripleo: Custom TripleO puppet manifests to deploy the overcloud.
  • tripleo-puppet-elements: third-party puppet modules. (Not used in this tutorial, but needs to be used for adding additional puppet modules.)

Gerrit patches used in this example

The gerrit patches used to describe this walkthrough are:

Changes prerequisites

The controller services are defined and configured via Heat resource chains. In the proposed patch ( controller services will be wired to a new Heat feature that allows it to dynamically include a set of nested stacks representing individual services via a Heat resource chain. The current example will use this interface to decompose the controller role into isolated services.


Additional Heat features must be defined in order to apply the same behaviour to other roles (compute, storage). Current work defined in:

Updating tripleo-heat-templates

This section will describe the changes needed for tripleo-heat-templates.

Folder structure convention for tripleo-heat-templates

Services should be defined in the services folder, depending on the service purpose.

  services          ---> To host all services.
    <service type>             ---> Folder to store a specific type services (If time, will store time based services like: NTP, timezone, Chrony among others).
      <service name>.yaml      ---> Main service manifest.
      <service name>-base.yaml ---> Main service configuration file.


No puppet manifests may be defined in the THT repository, they should go to the puppet-tripleo repository instead.


The use of a base class (<service>-base.yaml) is necessary in cases where a given ‘service’ (e.g. “heat”) is comprised of a number of individual component services (e.g. heat-api, heat-engine) which need to share some of the base configuration (such as rabbit credentials). Using a base class in those cases means we don’t need to duplicate that configuration. Refer to: for further details. Also, refer to duplicated-parameters for an use-case description.

Changes list

The list of changes in THT are:

  • If existing puppet modules for the added feature are present in tripleo-heat-templates, remove them, as they will be reintroduced in puppet-tripleo.
  • Create a service type specific folder in the root services folder (puppet/services/time).
  • Create a main service file inside the puppet/services folder (puppet/services/time/ntp.yaml).
  • Create a main configuration file linked to the main service file in the same folder (puppet/services/time/ntp-base.yaml).

Step 1 - Updating puppet references

Remove all puppet references for the composable service from the current manifests (*.pp). All the puppet logic will live in the puppet-tripleo repository based on a configuration step, so it is mandatory to remove all the puppet references from tripleo-heat-templates.

The updated .pp files for the NTP example were:

  • puppet/manifests/overcloud_cephstorage.pp
  • puppet/manifests/overcloud_compute.pp
  • puppet/manifests/overcloud_controller.pp
  • puppet/manifests/overcloud_controller_pacemaker.pp
  • puppet/manifests/overcloud_object.pp
  • puppet/manifests/overcloud_volume.pp

Step 2 - overcloud-resource-registry-puppet.yaml resource registry changes

The resource OS::TripleO::Services::Ntp must be defined in the resource registry (overcloud-resource-registry-puppet.yaml)

Create a new resource to configure your service using the folder naming convention.

Update this file as:

# services
OS::TripleO::Services: puppet/services/services.yaml
OS::TripleO::Services::Keystone: puppet/services/keystone.yaml
OS::TripleO::Services::GlanceApi: puppet/services/glance-api.yaml
OS::TripleO::Services::GlanceRegistry: puppet/services/glance-registry.yaml
OS::TripleO::Services::Ntp: puppet/services/time/ntp.yaml  ---> Your new service definition (Link to the main service definition file).

By updating the resource registry we are forcing to use a nested template to configure our resources. In the example case the created resource (OS::TripleO::Services::Ntp), will point to the corresponding service yaml file (puppet/services/time/ntp.yaml).

Step 3 - overcloud.yaml initial changes

It is mandatory to link our new feature to the ControllerServices parameter, which defines all the services enabled by default in the controller(s).

From overcloud.yaml in the parameters section find:

    - OS::TripleO::Services::Keystone
    - OS::TripleO::Services::GlanceApi
    - OS::TripleO::Services::GlanceRegistry
    - OS::TripleO::Services::Ntp              ---> New service deployed in the controller overcloud
  description: A list of service resources (configured in the Heat
               resource_registry) which represent nested stacks
               for each service that should get installed on the Controllers.
  type: comma_delimited_list

Update this section with your new service to be deployed to the controllers in the overcloud.

The parameter ControllerServices will be used by the ControllerServiceChain resource as follows:

  type: OS::TripleO::Services
    Services: {get_param: ControllerServices}
    EndpointMap: {get_attr: [EndpointMap, endpoint_map]}
    MysqlVirtualIPUri: {get_attr: [VipMap, net_ip_uri_map, {get_param: [ServiceNetMap, MysqlNetwork]}]}


Only for the controller role is available the deployment of services using the composable services approach, new Heat features must be defined in order to apply the same behaviour to other roles (compute, storage). Current work defined in:

Step 4 - Create the services yaml files

Create: puppet/services/time/ntp.yaml

This file will have all the configuration details for the service to be configured.

heat_template_version: 2016-04-08

description: >
NTP service deployment using puppet, this YAML file creates the interface between the HOT template and the puppet manifest that actually installs and configure NTP.

default: {} description: Mapping of service endpoint -> protocol. Typically set

via parameter_defaults in the resource registry.

type: json

default: [‘’,’’] description: NTP servers type: comma_delimited_list
default: [‘’] description: Listening interfaces type: comma_delimited_list

description: Role ntp using composable services. value:

tripleo::profile::base::time::ntp::ntpservers: {get_param: NtpServers} tripleo::profile::base::time::ntp::ntpinterfaces: {get_param: NtpInterfaces} tripleo::profile::base::time::ntp::step: ‘2’
step_config: |
include ::tripleo::profile::base::time::ntp


If it is needed, the templates can be decomposed to remove duplicated parameters among different deployment environments (i.e. using pacemaker). To do this follow to the section duplicated-parameters.

Updating puppet-tripleo

The puppet manifests that currently define overcloud node configuration are moved from the tripleo-heat-templates to new puppet-tripleo class definitions as part of the composable roles approach. In next iterations, all roles configurations should be moved also to puppet-tripleo. This section considers the addition of the ntp definition to puppet-tripleo.

Folder structure convention

Services should be defined in the services folder, depending on the service purpose.

  profile/base      ---> To host all services not using pacemaker.
    time            ---> Specific folder for time services (NTP, timezone, Chrony among others).
      ntp.pp        ---> Puppet manifest to configure the service.


For further information related to the current folders manifests structure refer to the puppet-tripleo repository.

Adding the puppet manifest

This step will reference how the puppet logic should be organized in puppet-tripleo.

Inside the manifests folder, add the service manifest following the folder structure (manifests/profile/base/time/ntp.pp) as:

class tripleo::profile::base::time::ntp (
  #We get the configuration step in which we can choose which steps to execute
  $step          = hiera('step'),
  $ntpservers    = [],
  $ntpinterfaces = [],
) {
  #step assigned for core modules.
  #(Check for further referencies about the configuration steps)
  if ($step >= 2){
    #We will call the NTP puppet class and assign our configuration values.
    #If needed additional Puppet packages can be added/installed by using the repo tripleo-puppet-elements
    if count($ntpservers) > 0 {
      class { '::ntp':
        servers    => $ntpservers,
        interfaces => $ntpinterfaces,

If users have followed all the previous steps, they should be able to configure their services using the composable roles guidelines.

Deployment tips for puppet-tripleo

This section will describe different ways of debugging puppet-tripleo changes.

Deploying puppet-tripleo using gerrit patches or source code repositories

When adding new features to THT, in some cases dependencies should be merged first in order to test newer patches, with the following procedure, the user will be able to create the overcloud images using WorkInProgress patches from gerrit code review without having them merged (for CI testing purposes).

If using third party repos included in the overcloud image, like i.e. the puppet-tripleo repository, your changes will not be available by default in the overcloud until you write them in the overcloud image (by default is: overcloud-full.qcow2)

In order to make quick changes to the overcloud image for testing purposes, you can:

Create the overcloud image by following (in_progress_review) :

export DIB_INSTALLTYPE_puppet_tripleo=source

export DIB_REPOLOCATION_puppet_tripleo=

export DIB_REPOREF_puppet_tripleo=refs/changes/25/310725/14

In order to avoid noise on IRC, it is possible to clone puppet-tripleo and apply the changes from your github account. In some cases this is particularly useful as there is no need to update the patchset number.

export DIB_INSTALLTYPE_puppet_tripleo=source

export DIB_REPOLOCATION_puppet_tripleo=<usergoeshere>/puppet-tripleo

Remove previously created images from glance and from the user home folder by:

rm -rf overcloud-full.*

glance image-delete overcloud-full

glance image-delete overcloud-full-initrd

glance image-delete overcloud-full-vmlinuz

After this step the images can be recreated by executing:

./tripleo-ci/scripts/ --overcloud-images

Pushing changes directly without re-creating the overcloud images

On the undercloud as the stack user, clone the openstack modules using:

set -eu
cd ~
if [ ! -e ~/puppet-modules ]; then
  mkdir ~/puppet-modules
  pushd ~/puppet-modules
    curl -O
    mapfile -t REPO_CLONES < <(cat source-repository-puppet-modules | awk '{ print $4 " " $3 }' | sed -e 's|/opt/stack/puppet-modules/||')
    for CLONE in "${REPO_CLONES[@]}"; do
      git clone $CLONE

Then, using the following script, upload the cloned puppet modules to swift

set -eu
cd ~
tar --exclude-vcs --show-transformed-names --transform 's|puppet-modules/|opt/stack/puppet-modules/|' -czvf puppet-modules.tar.gz puppet-modules
upload-swift-artifacts -f puppet-modules.tar.gz --environment puppet-modules.yaml

This will generates the puppet-modules.yaml environment file, which can be added to the deployment command (“-e puppet-modules.yaml”) and use cloned puppet modules

After every manual change to any of the puppet modules it is possible to re-run the upload to swift, and re-run overcloud deploy with the new env file. This process will speed up the changes in the ovecloud images.

Debugging puppet-tripleo from overcloud images

For debugging purposes, it is possible to mount the overcloud .qcow2 file:

#Install the libguest tool:
sudo yum install -y libguestfs-tools

#Create a temp folder to mount the overcloud-full image:
mkdir /tmp/overcloud-full

#Mount the image:
guestmount -a overcloud-full.qcow2 -i --rw /tmp/overcloud-full

#Check and validate all the changes to your overcloud image, go to /tmp/overcloud-full:
#  For example, in this step you can go to /opt/puppet-modules/tripleo,

#Umount the image
sudo umount /tmp/overcloud-full

From the mounted image file it is also possible to run, for testing purposes, the puppet manifests by using puppet apply and including your manifests:

sudo puppet apply -v --debug --modulepath=/tmp/overcloud-full/opt/stack/puppet-modules -e "include ::tripleo::services::time::ntp"

THT design patterns

Duplicated parameters

Problem: Having duplicated parameters in yaml files using the same <service>-base.yaml file. Example: MongoDB service using rplset name with pacemaker and without pacemaker.

This pattern will describe how to avoid duplicated parameters in the THT yaml files.

mongodb-base.yaml: This file should have all the common parameters between the different environments (With pacemaker and without pacemaker).

heat_template_version: 2016-04-08
description: >
  Configuration details for MongoDB service using composable roles
    type: string
    default: "tripleo"
    description: Additional parameters referenced outside the base file
      rplset_name: {get_param: MongoDbReplset}

In this way we will be able to reference the common parameter among all the yaml files requiring it.

Referencing the common parameter:

mongodb.yaml: Will have specific parameters to deploy mongodb without pacemaker.

heat_template_version: 2016-04-08
description: >
  MongoDb service deployment using puppet
    type: ./mongodb-base.yaml
    description: Service mongodb using composable services.
          - tripleo::profile::base::database::mongodb::mongodb_replset: {get_attr: [MongoDbBase, aux_parameters, rplset_name]}
      step_config: |
        include ::tripleo::profile::base::database::mongodb

In this case mongodb.yaml can use this common parameter, and assign it to a specific puppet manifest parameter depending on the chosen environment.

If using the parameter ‘EndpointMap’ in the base file, is mandatory to passing it from the service file.

In the service file:

    default: {}
    description: Mapping of service endpoint -> protocol. Typically set
                 via parameter_defaults in the resource registry.
    type: json
    type: ./<ServiceName>-base.yaml
      EndpointMap: {get_param: EndpointMap}

This will pass the endpoint information to the base config file.