Skip to content

Terraform - Headstart with Google Cloud Platform

Overview

This document to demonstrate concepts configuration project build in Google Cloud Platform with a minimal in mind.

Requirements

The project will deploy following components:

Start

# Organization Roles
# This required organization admin
roles = (
    billing.admin
    resourcemanager.organizationAdmin
    resourcemanager.folderCreator
    resourcemanager.projectCreator
)
for role in "${roles[@]}"
do
  gcloud organizations add-iam-policy-binding $ORGANIZATION_ID \
      --member=user:<email></email> \
      --role=roles/$role
done;

# -----------------------------------------------------
# Module: Central Artifact
# -----------------------------------------------------

# Description:
# Create repository for Docker images in project

# Note:
# Artifact is next-generation of Container Repository

# -----------------------------------------------------
# Repository of images for data pipelines
# -----------------------------------------------------
resource "google_artifact_registry_repository" "data-team" {
  location      = var.project_region
  project       = var.project_id
  repository_id = "artifact-registry"
  description   = "Centralized repository of images for data pipelines"
  format        = "DOCKER"
  labels = {
    team       = "data"
    managed_by = "terraform"
  }
}

Terraform with Google Cloud Platform

Best practices for Terraform [GCP] https://cloud.google.com/docs/terraform/best-practices-for-terraform

Identities Synponis for Users, Groups, Accounts

Each entry can have one of the following values:

[a] user:{emailid}: An email address that represents a specific Google account. For example, alice@gmail.com or joe@example.com.

[b] serviceAccount:{emailid}: An email address that represents a service account. For example, my-other-app@appspot.gserviceaccount.com.

[c] group:{emailid}: An email address that represents a Google group. For example, admins@example.com.

[d] domain:{domain}: A G Suite domain (primary, instead of alias) name that represents all the users of that domain. For example, google.com or example.com.

Non-version bucket

Some bucket requrieed to have non-version buckets

resource "google_storage_bucket" "required_non_version_bucket_name" {
  name          = "required-non-version-bucket-name"
  project       = var.project_id
  force_destroy = false
  location      = upper("${var.project_region}")
  storage_class = "STANDARD"

  versioning {
    enabled = false
  }

  labels = {
    type            = "infrastructure"
    managed_by      = "terraform"
    required_no_ttl = "true"
  }
}

Permission of Terrafrom

Default Service Account <project_number>-compute@developer.gserviceaccount.com < Default SA

The SA has required following permissions:

- dataproc.agents.create,
- dataproc.agents.delete,
- dataproc.agents.get,
- dataproc.agents.update,
- dataproc.tasks.lease,
- dataproc.tasks.listInvalidatedLeases,
- dataproc.tasks.reportStatus

This has been included in "roles/dataproc.worker"

Service Accounts must have either 'Dataproc/Dataproc Worker' role or all permissions granted by the role.

See https://cloud.google.com/dataproc/docs/concepts/iam for further details.

  • Requested image requires minimum boot disk size of 30 GB; requested 15 GB

For the service agents

roles/iam.serviceAccountUser roles/buckets/list

Reference

Appendix