Service deployment on Nomad POC by LDiazN · Pull Request #461 · ooni/devops

LDiazN · 2026-07-27T15:23:40Z

This PR is a POC to show how to use Nomad to deploy our API services, in the same spirit as #454

Background

Nomad is a simple orchestrator to run (mostly) containerized services. It runs works with an "Admin" server that allocates tasks to "client" servers. It's a single binary and very easy to setup.

Installing Nomad on a server in "Client" mode makes it a pool of resources where tasks can be allocated by an "Admin" server

It uses the same HCL language as Terraform, it integrates well with Terraform

This PR

installed this on a stale host that we no longer use (anonc.dev.ooni.io)
Made an ansible role that:
- Installs Nomad
- Installs and configures Docker and Podman
- Creates the nginx rules to access the Nomad UI and routes API traffic
Adds Nomad to Terraform
Adds template job definition for API jobs. Adding a new service means instantiating the template with different config values and env variables
Adds a valkey job

Highlights

Service configuration managed through terraform: helpful to wire it to AWS resources
Our docker images don't need any change, they are fully compatible with Nomad
Adding jobs means filling simple configuration values (port, memory, docker image), secrets and environment variables
The UI is very comprehensive and responsive: We can get logs and log into the container using the web UI
It uses canary deployment for zero downtime deployment
Since we don't use autoscaling, services have a fixed port. Nginx maps urls by path to the correct service, similar to how we use the AWS load balancer
Nomad supports specifying an URL for health checking, also similar to what we do on AWS
Easy to spin up new services besides the API itself, like Valkey
Documentation was very useful and comprehensive
Services can be stopped and services with a single button in the Web UI

Cons

If we decide to use autoscaling in the future, it requires yet another process running for service discovery: Consul
Setting up Podman for rootless containers was painful
CI/CD is involved; the running container image is determined by the tag in terraform. This means that the CI has to write the tag into a tfvars in the devops repo and then run terraform -targeted to this service
Nomad uses raft for consensus between nodes, which can be hard to maintain
The metrics page for each service instance is broken, but the AWS one wasn't very helpful either. Since we control the host node and it runs over Docker/Podman, we can install a Prometheus exporter for container metrics that would be more useful that this page even if it was working
Podman has support via a plugin. Works well but it doesn't comes out of the box like Docker

github-actions · 2026-07-27T15:24:41Z

Ansible Run Output 🤖

Ansible Playbook Recap 🔍

Ansible playbook output 📖`success`

Show Execution


$ ansible-playbook playbook.yml --check --diff -i ../tf/modules/ansible_inventory/inventories/inventory-dev.ini
[WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match 'all'
[ERROR]: The role 'geerlingguy.docker' was not found in: /home/runner/work/devops/devops/ansible/roles:/home/runner/.ansible/roles:/usr/share/ansible/roles:/etc/ansible/roles:/home/runner/work/devops/devops/ansible
Origin: /home/runner/work/devops/devops/ansible/deploy-testlists.yml:16:7

14         node_exporter_host: "0.0.0.0"
15         node_exporter_options: ""
16     - role: geerlingguy.docker
         ^ column 7


Pusher	@LDiazN
Action	pull_request
Working Directory
Workflow	.github/workflows/check_ansible.yml
Last updated	Mon, 27 Jul 2026 15:27:08 GMT

github-actions · 2026-07-27T15:24:46Z

Terraform Run Output 🤖

Format and Style 🖌`failure`

Initialization ⚙️`success`

Validation 🤖`success`

Validation Output


$ terraform validate

Warning: Argument is deprecated

  with aws_s3_bucket.anoncred_manifests,
  on main.tf line 285, in resource "aws_s3_bucket" "anoncred_manifests":
 285: resource "aws_s3_bucket" "anoncred_manifests" {

Use the aws_s3_bucket_versioning resource instead

Warning: Deprecated value used

  on ../../modules/ecs_cluster/outputs.tf line 22, in output "container_host_role":
  22:   value = aws_iam_role.container_host

  The deprecation originates from module.ooniapi_cluster.aws_iam_role.container_host.inline_policy

Deprecated resource block "inline_policy" used. Refer to the provider
documentation for details.

(and 3 more similar warnings elsewhere)

Warning: Available Write-only Attribute Alternative

  with module.ooni_monitoring.aws_ssm_parameter.ooni_monitoring_access_key,
  on ../../modules/ooni_monitoring/main.tf line 47, in resource "aws_ssm_parameter" "ooni_monitoring_access_key":
  47:   value = aws_iam_access_key.ooni_monitoring.id

The attribute value has a write-only alternative value_wo available. Use the
write-only alternative of the attribute when possible.

(and one more similar warning elsewhere)
Success! The configuration is valid, but there were some validation warnings
as shown above.

Plan 📖`success`

undefined

Show Plan


$ terraform plan

Warning: Deprecated Parameter

The parameter "dynamodb_table" is deprecated. Use parameter "use_lockfile"
instead.
module.ansible_inventory.local_file.ansible_inventory: Refreshing state... [id=b6de844ed8d384f890fa6f467502390de843f758]
module.adm_iam_roles.tls_private_key.oonidevops: Refreshing state... [id=b49a9fdb9f720320340226016efe24808dd68203]
module.ooniapi_frontend.random_id.artifact_id: Refreshing state... [id=_3cQlA]
random_id.artifact_id: Refreshing state... [id=8Ujqew]
module.ooniapi_testlists.data.cloudinit_config.ooni_ec2: Reading...
data.dns_a_record_set.monitoring_host: Reading...
module.ooni_jumphost.data.cloudinit_config.ooni_ec2: Reading...
module.ooni_anonc.data.cloudinit_config.ooni_ec2: Reading...
module.ooni_jumphost.data.cloudinit_config.ooni_ec2: Read complete after 0s [id=2022394177]
module.ooni_fastpath.module.ec2.data.cloudinit_config.ooni_ec2: Reading...
module.ansible_inventory.null_resource.ansible_update_known_hosts: Refreshing state... [id=236461505953331670]
module.ooniapi_testlists.data.cloudinit_config.ooni_ec2: Read complete after 0s [id=2022394177]
module.ooni_fastpath.module.ec2.data.cloudinit_config.ooni_ec2: Read complete after 0s [id=2022394177]
module.ooni_clickhouse_proxy.data.cloudinit_config.ooni_ec2: Reading...
module.ooni_anonc.data.cloudinit_config.ooni_ec2: Read complete after 0s [id=2022394177]
module.ooni_monitoring_proxy.data.cloudinit_config.ooni_ec2: Reading...
module.ooni_monitoring_proxy.data.cloudinit_config.ooni_ec2: Read complete after 0s [id=2022394177]
module.ooni_clickhouse_proxy.data.cloudinit_config.ooni_ec2: Read complete after 0s [id=2022394177]
data.dns_a_record_set.monitoring_host: Read complete after 0s [id=monitoring.ooni.org]
data.aws_availability_zones.available: Reading...
module.ooniapi_oonifindings.data.aws_ecs_container_definition.ooniapi_service_current[0]: Reading...
module.oonidevops_github_user.aws_iam_policy.oonidevops_github: Refreshing state... [id=arn:aws:iam::905418398257:policy/oonidevops-github-policy]
data.aws_ssm_parameter.nomad_address: Reading...
aws_s3_bucket.ooni_private_config_bucket: Refreshing state... [id=ooni-config-eu-central-1-f148ea7b]
module.ooni_monitoring_proxy.data.aws_ssm_parameter.ubuntu_22_ami: Reading...
data.aws_ssm_parameter.anonc_secret_key: Reading...
module.ooni_anonc.data.aws_ssm_parameter.ubuntu_22_ami: Reading...
module.ooniapi_frontend.aws_s3_bucket.athena_results: Refreshing state... [id=ooni-athena-results-ff771094]
aws_s3_bucket.anoncred_manifests: Refreshing state... [id=ooni-anoncreds-manifests-dev-eu-central-1]
module.ooniapi_oonifindings.aws_iam_role.ooniapi_service_task: Refreshing state... [id=ooniapi-service-oonifindings-task-role]
module.testlists_builder.aws_iam_policy.codepipeline: Refreshing state... [id=arn:aws:iam::905418398257:policy/service-role/codepipeline-oonidkr-testlists]
module.oonidevops_github_user.aws_iam_user.oonidevops_github: Refreshing state... [id=oonidevops-github]
data.aws_ssm_parameter.clickhouse_oonimeasurements_test_url: Reading...
module.ooni_monitoring_proxy.data.aws_ssm_parameter.ubuntu_22_ami: Read complete after 1s [id=/aws/service/canonical/ubuntu/server/22.04/stable/current/amd64/hvm/ebs-gp2/ami-id]
aws_s3_bucket.ooniapi_codepipeline_bucket: Refreshing state... [id=codepipeline-ooniapi-eu-central-1-f148ea7b]
data.aws_ssm_parameter.anonc_secret_key: Read complete after 1s [id=/oonidevops/secrets/zkp/secret_key]
data.aws_ssm_parameter.nomad_token: Reading...
module.ooni_anonc.data.aws_ssm_parameter.ubuntu_22_ami: Read complete after 1s [id=/aws/service/canonical/ubuntu/server/22.04/stable/current/amd64/hvm/ebs-gp2/ami-id]
module.oonidevops_github_user.aws_secretsmanager_secret.oonidevops_github: Refreshing state... [id=arn:aws:secretsmanager:eu-central-1:905418398257:secret:oonidevops/github_user/access_key_json-9JTJgd]
data.aws_ssm_parameter.nomad_address: Read complete after 1s [id=/oonidevops/secrets/nomad_address]
module.ooniapi_oonifindings.aws_cloudwatch_log_group.ooniapi_service: Refreshing state... [id=ooni-ecs-group/ooniapi-service-oonifindings]
data.aws_availability_zones.available: Read complete after 1s [id=eu-central-1]
module.adm_iam_roles.aws_key_pair.oonidevops: Refreshing state... [id=oonidevops]
module.ooniapi_oonifindings.data.aws_ecs_container_definition.ooniapi_service_current[0]: Read complete after 1s [id=ooniapi-service-oonifindings-td/ooniapi-service-oonifindings]
module.ooniapi_oonirun_deployer.aws_iam_policy.codepipeline: Refreshing state... [id=arn:aws:iam::905418398257:policy/service-role/codepipeline-ooniapi-oonirun]
data.aws_ssm_parameter.clickhouse_oonimeasurements_test_url: Read complete after 0s [id=/oonidevops/secrets/clickhouse_oonimeasurements_test_url]
module.ooniapi_reverseproxy.aws_cloudwatch_log_group.ooniapi_service: Refreshing state... [id=ooni-ecs-group/ooniapi-service-reverseproxy]
data.aws_ssm_parameter.nomad_token: Read complete after 0s [id=/oonidevops/secrets/nomad_token]
data.aws_ssm_parameter.clickhouse_oonimeasurements_url: Reading...
module.testlists_builder.data.aws_caller_identity.current: Reading...
data.aws_ssm_parameter.oonipg_url: Reading...
module.testlists_builder.data.aws_caller_identity.current: Read complete after 0s [id=905418398257]
module.ooniapi_ooniprobe.aws_cloudwatch_log_group.ooniapi_service: Refreshing state... [id=ooni-ecs-group/ooniapi-service-ooniprobe]
data.aws_ssm_parameter.clickhouse_oonimeasurements_url: Read complete after 0s [id=/oonidevops/secrets/clickhouse_oonimeasurements_url]
module.ooniapi_ooniauth.aws_cloudwatch_log_group.ooniapi_service: Refreshing state... [id=ooni-ecs-group/ooniapi-service-ooniauth]
data.aws_ssm_parameter.oonipg_url: Read complete after 0s [id=/oonidevops/secrets/ooni-tier0-postgres/postgresql_write_url]
data.aws_ssm_parameter.jwt_secret: Reading...
data.aws_ssm_parameter.jwt_secret: Read complete after 0s [id=/oonidevops/secrets/ooni_services/jwt_secret]
module.ooniapi_frontend.aws_s3_bucket.load_balancer_logs: Refreshing state... [id=lb-logs-eu-central-1-ff771094]
module.ooniapi_cluster.aws_iam_role.container_host: Refreshing state... [id=ooniapi-ecs-cluster-container-host-role]
module.ooniapi_cluster.data.aws_ssm_parameter.ecs_optimized_ami: Reading...
module.ooni_monitoring.aws_iam_user.ooni_monitoring: Refreshing state... [id=oonidevops-monitoring]
module.ooniapi_reverseproxy.data.aws_ecs_container_definition.ooniapi_service_current[0]: Reading...
data.aws_ssm_parameter.clickhouse_oonirun_url: Reading...
module.ooniapi_ooniprobe.aws_iam_role.ooniapi_service_task: Refreshing state... [id=ooniapi-service-ooniprobe-task-role]
module.ooniapi_user.aws_secretsmanager_secret.aws_access_key_id: Refreshing state... [id=arn:aws:secretsmanager:eu-central-1:905418398257:secret:oonidevops/ooniapi_user/aws_access_key_id-EcXOBx]
module.ooniapi_cluster.data.aws_ssm_parameter.ecs_optimized_ami: Read complete after 0s [id=/aws/service/ecs/optimized-ami/amazon-linux-2023/recommended]
module.oonitier1plus_cluster.aws_iam_role.container_host: Refreshing state... [id=oonitier1plus-ecs-cluster-container-host-role]
module.ooniapi_reverseproxy.data.aws_ecs_container_definition.ooniapi_service_current[0]: Read complete after 0s [id=ooniapi-service-reverseproxy-td/ooniapi-service-reverseproxy]
module.ooniapi_reverseproxy.aws_iam_role.ooniapi_service_task: Refreshing state... [id=ooniapi-service-reverseproxy-task-role]
data.aws_ssm_parameter.clickhouse_oonirun_url: Read complete after 0s [id=/oonidevops/secrets/clickhouse_oonirun_url]
aws_iam_role.monitoring_proxy_yace: Refreshing state... [id=monitoring-proxy-yace]
aws_acm_certificate.ooniapi_frontend: Refreshing state... [id=arn:aws:acm:eu-central-1:905418398257:certificate/c6d8f7a1-5b6d-4302-9aa6-92850ded970e]
module.ooniapi_oonirun.data.aws_ecs_container_definition.ooniapi_service_current[0]: Reading...
module.adm_iam_roles.data.aws_iam_policy_document.assume_role: Reading...
module.adm_iam_roles.data.aws_iam_policy_document.assume_role: Read complete after 0s [id=3569030007]
data.aws_ssm_parameter.jwt_secret_legacy: Reading...
aws_secretsmanager_secret.oonipg_url: Refreshing state... [id=arn:aws:secretsmanager:eu-central-1:905418398257:secret:oonidevops/ooni-tier0-postgres/postgresql_url-w62CTZ]
module.ooniapi_oonirun.aws_iam_role.ooniapi_service_task: Refreshing state... [id=ooniapi-service-oonirun-task-role]
module.ooniapi_oonirun.data.aws_ecs_container_definition.ooniapi_service_current[0]: Read complete after 0s [id=ooniapi-service-oonirun-td/ooniapi-service-oonirun]
module.ooniapi_ooniauth_deployer.aws_iam_policy.codepipeline: Refreshing state... [id=arn:aws:iam::905418398257:policy/service-role/codepipeline-ooniapi-ooniauth]
data.aws_ssm_parameter.jwt_secret_legacy: Read complete after 0s [id=/oonidevops/secrets/ooni_services/jwt_secret_legacy]
module.ooniapi_oonimeasurements_deployer.aws_iam_policy.codepipeline: Refreshing state... [id=arn:aws:iam::905418398257:policy/service-role/codepipeline-ooniapi-oonimeasurements]
module.adm_iam_roles.aws_iam_policy.oonidevops: Refreshing state... [id=arn:aws:iam::905418398257:policy/OONIDevopsPolicy]
module.ooniapi_oonimeasurements.aws_cloudwatch_log_group.ooniapi_service: Refreshing state... [id=ooni-ecs-group/ooniapi-service-oonimeasurements]
module.ooniapi_oonimeasurements.aws_iam_role.ooniapi_service_task: Refreshing state... [id=ooniapi-service-oonimeasurements-task-role]
module.ooni_jumphost.data.aws_ssm_parameter.ubuntu_22_ami: Reading...
module.ooni_fastpath.module.ec2.data.aws_ssm_parameter.ubuntu_22_ami: Reading...
aws_s3_bucket.ooniprobe_failed_reports: Refreshing state... [id=ooniprobe-failed-reports-eu-central-1]
module.ooni_fastpath.module.ec2.data.aws_ssm_parameter.ubuntu_22_ami: Read complete after 0s [id=/aws/service/canonical/ubuntu/server/22.04/stable/current/amd64/hvm/ebs-gp2/ami-id]
module.ooniapi_ooniauth.data.aws_ecs_container_definition.ooniapi_service_current[0]: Reading...
module.ooni_jumphost.data.aws_ssm_parameter.ubuntu_22_ami: Read complete after 0s [id=/aws/service/canonical/ubuntu/server/22.04/stable/current/amd64/hvm/ebs-gp2/ami-id]
module.ooniapi_user.aws_ses_email_identity.ooniapi: Refreshing state... [id=admin+dev@ooni.org]
module.ooniapi_ooniauth.aws_iam_role.ooniapi_service_task: Refreshing state... [id=ooniapi-service-ooniauth-task-role]
module.fastpath_builder.aws_iam_policy.codepipeline: Refreshing state... [id=arn:aws:iam::905418398257:policy/service-role/codepipeline-oonidkr-fastpath]
module.oonitier1plus_cluster.aws_cloudwatch_log_group.ooniapi_services: Refreshing state... [id=ooni-ecs-group/oonitier1plus-ecs-cluster]
module.ooniapi_ooniauth.data.aws_ecs_container_definition.ooniapi_service_current[0]: Read complete after 0s [id=ooniapi-service-ooniauth-td/ooniapi-service-ooniauth]
module.ooniapi_reverseproxy_deployer.data.aws_caller_identity.current: Reading...
module.ooniapi_ooniprobe_deployer.data.aws_caller_identity.current: Reading...
module.fastpath_builder.data.aws_caller_identity.current: Reading...
module.ooniapi_oonifindings_deployer.aws_iam_policy.codepipeline: Refreshing state... [id=arn:aws:iam::905418398257:policy/service-role/codepipeline-ooniapi-oonifindings]
module.ooni_clickhouse_proxy.data.aws_ssm_parameter.ubuntu_22_ami: Reading...
module.ooniapi_reverseproxy_deployer.data.aws_caller_identity.current: Read complete after 0s [id=905418398257]
data.aws_ssm_parameter.clickhouse_ooniprobe_url: Reading...
module.ooniapi_testlists.data.aws_ssm_parameter.ubuntu_22_ami: Reading...
module.ooni_clickhouse_proxy.data.aws_ssm_parameter.ubuntu_22_ami: Read complete after 0s [id=/aws/service/canonical/ubuntu/server/22.04/stable/current/amd64/hvm/ebs-gp2/ami-id]
module.ooniapi_oonirun_deployer.data.aws_caller_identity.current: Reading...
module.ooniapi_ooniprobe_deployer.data.aws_caller_identity.current: Read complete after 0s [id=905418398257]
module.ooniapi_oonirun.aws_cloudwatch_log_group.ooniapi_service: Refreshing state... [id=ooni-ecs-group/ooniapi-service-oonirun]
module.ooniapi_oonifindings_deployer.data.aws_caller_identity.current: Reading...
data.aws_ssm_parameter.clickhouse_ooniprobe_url: Read complete after 0s [id=/oonidevops/secrets/clickhouse_ooniprobe_url]
module.ooniapi_ooniauth_deployer.data.aws_caller_identity.current: Reading...
module.ooniapi_testlists.data.aws_ssm_parameter.ubuntu_22_ami: Read complete after 0s [id=/aws/service/canonical/ubuntu/server/22.04/stable/current/amd64/hvm/ebs-gp2/ami-id]
module.ooniapi_reverseproxy_deployer.aws_iam_policy.codepipeline: Refreshing state... [id=arn:aws:iam::905418398257:policy/service-role/codepipeline-ooniapi-reverseproxy]
module.fastpath_builder.data.aws_caller_identity.current: Read complete after 0s [id=905418398257]
module.ooniapi_user.aws_iam_user.ooniapi: Refreshing state... [id=oonidevops-ooniapi]
module.ooniapi_oonimeasurements_deployer.data.aws_caller_identity.current: Reading...
data.aws_ssm_parameter.clickhouse_readonly_url_test: Reading...
module.adm_iam_roles.aws_secretsmanager_secret.oonidevops_deploy_key: Refreshing state... [id=arn:aws:secretsmanager:eu-central-1:905418398257:secret:oonidevops/deploy_key-2ebqSe]
module.ooniapi_oonirun_deployer.data.aws_caller_identity.current: Read complete after 1s [id=905418398257]
module.ooniapi_ooniprobe.data.aws_ecs_container_definition.ooniapi_service_current[0]: Reading...
data.aws_ssm_parameter.do_token: Reading...
data.aws_ssm_parameter.account_id_hashing_key: Reading...
data.aws_ssm_parameter.clickhouse_readonly_url_test: Read complete after 0s [id=/oonidevops/secrets/clickhouse_readonly_test_url]
aws_s3_bucket.oonith_codepipeline_bucket: Refreshing state... [id=codepipeline-oonith-eu-central-1-f148ea7b]
module.ooniapi_oonifindings_deployer.data.aws_caller_identity.current: Read complete after 1s [id=905418398257]
module.ooniapi_user.aws_secretsmanager_secret.aws_secret_access_key: Refreshing state... [id=arn:aws:secretsmanager:eu-central-1:905418398257:secret:oonidevops/ooniapi_user/aws_secret_access_key-L0DQDr]
module.ooniapi_ooniauth_deployer.data.aws_caller_identity.current: Read complete after 1s [id=905418398257]
data.aws_ssm_parameter.prometheus_metrics_password: Reading...
module.ooniapi_ooniprobe.data.aws_ecs_container_definition.ooniapi_service_current[0]: Read complete after 0s [id=ooniapi-service-ooniprobe-td/ooniapi-service-ooniprobe]
module.oonitier1plus_cluster.data.aws_ssm_parameter.ecs_optimized_ami: Reading...
module.ooniapi_oonimeasurements_deployer.data.aws_caller_identity.current: Read complete after 1s [id=905418398257]
module.ooniapi_ooniprobe_deployer.aws_iam_policy.codepipeline: Refreshing state... [id=arn:aws:iam::905418398257:policy/service-role/codepipeline-ooniapi-ooniprobe]
module.ooniapi_cluster.aws_cloudwatch_log_group.ooniapi_services: Refreshing state... [id=ooni-ecs-group/ooniapi-ecs-cluster]
data.aws_ssm_parameter.do_token: Read complete after 0s [id=/oonidevops/secrets/digitalocean_access_token]
module.ooniapi_oonifindings.aws_iam_role_policy.ooniapi_service_task: Refreshing state... [id=ooniapi-service-oonifindings-task-role:ooniapi-service-oonifindings-task-role]
module.testlists_builder.aws_iam_role.codepipeline: Refreshing state... [id=codepipeline-oonidkr-testlists]
data.aws_ssm_parameter.account_id_hashing_key: Read complete after 0s [id=/oonidevops/secrets/ooni_services/account_id_hashing_key]
module.oonidevops_github_user.aws_iam_access_key.oonidevops_github: Refreshing state... [id=AKIA5FTZELIYXDN55SMS]
module.oonidevops_github_user.aws_iam_user_policy_attachment.oonidevops_github: Refreshing state... [id=oonidevops-github-20240313195612421500000001]
module.oonitier1plus_cluster.data.aws_ssm_parameter.ecs_optimized_ami: Read complete after 0s [id=/aws/service/ecs/optimized-ami/amazon-linux-2023/recommended]
data.aws_ssm_parameter.prometheus_metrics_password: Read complete after 0s [id=/oonidevops/ooni_services/prometheus_metrics_password]
module.ooniapi_cluster.aws_iam_instance_profile.container_host: Refreshing state... [id=ooniapi-ecs-cluster]
module.ooniapi_oonirun_deployer.aws_iam_role.codepipeline: Refreshing state... [id=codepipeline-ooniapi-oonirun]
module.ooni_monitoring.aws_iam_user_policy.ooni_monitoring: Refreshing state... [id=oonidevops-monitoring:oonidevops-monitoring-policy]
module.ooniapi_cluster.aws_iam_role_policy.container_host: Refreshing state... [id=ooniapi-ecs-cluster-container-host-role:ooniapi-ecs-cluster-instance-role-policy]
module.ooni_monitoring.aws_iam_access_key.ooni_monitoring: Refreshing state... [id=AKIA5FTZELIYWULOT65S]
module.ooniapi_ooniprobe.aws_iam_role_policy.ooniapi_service_task: Refreshing state... [id=ooniapi-service-ooniprobe-task-role:ooniapi-service-ooniprobe-task-role]
module.oonitier1plus_cluster.aws_iam_role_policy.container_host: Refreshing state... [id=oonitier1plus-ecs-cluster-container-host-role:oonitier1plus-ecs-cluster-instance-role-policy]
module.oonitier1plus_cluster.aws_iam_instance_profile.container_host: Refreshing state... [id=oonitier1plus-ecs-cluster]
module.ooniapi_reverseproxy.aws_iam_role_policy.ooniapi_service_task: Refreshing state... [id=ooniapi-service-reverseproxy-task-role:ooniapi-service-reverseproxy-task-role]
aws_iam_instance_profile.monitoring_proxy_yace: Refreshing state... [id=oonidevops-dev-monitoring-proxy-yace]
aws_iam_role_policy.monitoring_proxy_yace: Refreshing state... [id=monitoring-proxy-yace:yace-cloudwatch-read]
module.ooniapi_ooniauth_deployer.aws_iam_role.codepipeline: Refreshing state... [id=codepipeline-ooniapi-ooniauth]
module.ooniapi_oonirun.aws_iam_role_policy.ooniapi_service_task: Refreshing state... [id=ooniapi-service-oonirun-task-role:ooniapi-service-oonirun-task-role]
module.ooniapi_oonimeasurements_deployer.aws_iam_role.codepipeline: Refreshing state... [id=codepipeline-ooniapi-oonimeasurements]
module.adm_iam_roles.aws_iam_role.oonidevops: Refreshing state... [id=oonidevops]
module.ooniapi_oonimeasurements.aws_iam_role_policy.ooniapi_service_task: Refreshing state... [id=ooniapi-service-oonimeasurements-task-role:ooniapi-service-oonimeasurements-task-role]
module.ooniapi_frontend.aws_s3_bucket_lifecycle_configuration.athena_results: Refreshing state... [id=ooni-athena-results-ff771094]
module.ooniapi_frontend.aws_athena_workgroup.ooni_workgroup: Refreshing state... [id=ooni-workgroup]
module.ooniapi_frontend.aws_athena_database.load_balancer_logs: Refreshing state... [id=load_balancer_logs]
aws_s3_object.test_manifest: Refreshing state... [id=test_manifest.json]
aws_s3_bucket_policy.anonc_manifests_policy: Refreshing state... [id=ooni-anoncreds-manifests-dev-eu-central-1]
aws_s3_object.manifest: Refreshing state... [id=manifest.json]
aws_s3_bucket_versioning.anoncred_manifests_version: Refreshing state... [id=ooni-anoncreds-manifests-dev-eu-central-1]
aws_s3_bucket_ownership_controls.anonc_manifests: Refreshing state... [id=ooni-anoncreds-manifests-dev-eu-central-1]
aws_s3_bucket_public_access_block.anonc_manifests: Refreshing state... [id=ooni-anoncreds-manifests-dev-eu-central-1]
module.fastpath_builder.aws_iam_role.codepipeline: Refreshing state... [id=codepipeline-oonidkr-fastpath]
module.ooniapi_ooniauth.aws_iam_role_policy.ooniapi_service_task: Refreshing state... [id=ooniapi-service-ooniauth-task-role:ooniapi-service-ooniauth-task-role]
aws_route53_record.ooniapi_frontend_cert_validation["ooniprobe.dev.ooni.io"]: Refreshing state... [id=Z055356431RGCLK3JXZDL__a064be8aa084a037ff9fa5e3e541c87d.ooniprobe.dev.ooni.io._CNAME]
aws_route53_record.ooniapi_frontend_cert_validation["oonimeasurements.dev.ooni.io"]: Refreshing state... [id=Z055356431RGCLK3JXZDL__8fb10887c4ca7af87e33703c03c4c82e.oonimeasurements.dev.ooni.io._CNAME]
aws_route53_record.ooniapi_frontend_cert_validation["oonirun.dev.ooni.io"]: Refreshing state... [id=Z055356431RGCLK3JXZDL__05c891caeb4509d4cd7f9c24d8b6dbd0.oonirun.dev.ooni.io._CNAME]
aws_route53_record.ooniapi_frontend_cert_validation["testlists.dev.ooni.io"]: Refreshing state... [id=Z055356431RGCLK3JXZDL__2b445be660c03a7eecbd5429f176ec99.testlists.dev.ooni.io._CNAME]
aws_route53_record.ooniapi_frontend_cert_validation["api.dev.ooni.io"]: Refreshing state... [id=Z055356431RGCLK3JXZDL__cd4729fc0c282e771d056e719a7bdf4f.api.dev.ooni.io._CNAME]
aws_route53_record.ooniapi_frontend_cert_validation["ooniauth.dev.ooni.io"]: Refreshing state... [id=Z055356431RGCLK3JXZDL__48cd4e71cee9930614228176b7deefb9.ooniauth.dev.ooni.io._CNAME]
aws_route53_record.ooniapi_frontend_cert_validation["8.th.dev.ooni.io"]: Refreshing state... [id=Z055356431RGCLK3JXZDL__ef17825e5fd9713f596344bdd9626f5e.8.th.dev.ooni.io._CNAME]
module.ooniapi_oonifindings_deployer.aws_iam_role.codepipeline: Refreshing state... [id=codepipeline-ooniapi-oonifindings]
module.oonitier1plus_cluster.aws_ecs_cluster.main: Refreshing state... [id=arn:aws:ecs:eu-central-1:905418398257:cluster/oonitier1plus-ecs-cluster]
module.ooniapi_user.aws_iam_user_policy.ooniapi: Refreshing state... [id=oonidevops-ooniapi:oonidevops-ooniapi-policy]
module.ooniapi_user.aws_iam_access_key.ooniapi: Refreshing state... [id=AKIA5FTZELIYSK2XEVOT]
module.ooniapi_reverseproxy_deployer.aws_iam_role.codepipeline: Refreshing state... [id=codepipeline-ooniapi-reverseproxy]
module.adm_iam_roles.aws_secretsmanager_secret_version.oonidevops_deploy_key: Refreshing state... [id=arn:aws:secretsmanager:eu-central-1:905418398257:secret:oonidevops/deploy_key-2ebqSe|terraform-20240925140131946100000002]
module.ooniapi_ooniprobe_deployer.aws_iam_role.codepipeline: Refreshing state... [id=codepipeline-ooniapi-ooniprobe]
module.oonidevops_github_user.aws_secretsmanager_secret_version.oonidevops_github: Refreshing state... [id=arn:aws:secretsmanager:eu-central-1:905418398257:secret:oonidevops/github_user/access_key_json-9JTJgd|terraform-20240519071250187000000004]
nomad_variable.oonimeasurements_secrets: Refreshing state... [id=nomad/jobs/oonimeasurements@default]
nomad_job.valkey: Refreshing state... [id=valkey]
module.ooni_monitoring.aws_ssm_parameter.ooni_monitoring_access_key: Refreshing state... [id=/oonidevops/secrets/ooni_monitoring/access_key]
module.ooni_monitoring.aws_ssm_parameter.ooni_monitoring_secret_key: Refreshing state... [id=/oonidevops/secrets/ooni_monitoring/secret_key]
module.ooniapi_cluster.aws_ecs_cluster.main: Refreshing state... [id=arn:aws:ecs:eu-central-1:905418398257:cluster/ooniapi-ecs-cluster]
module.ooniapi_frontend.aws_s3_bucket_policy.alb_logs_policy: Refreshing state... [id=lb-logs-eu-central-1-ff771094]
module.ooniapi_frontend.aws_s3_bucket_ownership_controls.load_balancer_logs: Refreshing state... [id=lb-logs-eu-central-1-ff771094]
module.ooniapi_frontend.aws_s3_bucket_lifecycle_configuration.load_balancer_logs: Refreshing state... [id=lb-logs-eu-central-1-ff771094]
aws_s3_bucket_acl.anonc_manifests: Refreshing state... [id=ooni-anoncreds-manifests-dev-eu-central-1,public-read]
module.ooniapi_user.aws_secretsmanager_secret_version.aws_access_key_id: Refreshing state... [id=arn:aws:secretsmanager:eu-central-1:905418398257:secret:oonidevops/ooniapi_user/aws_access_key_id-EcXOBx|terraform-20240314200140918400000007]
module.ooniapi_user.aws_secretsmanager_secret_version.aws_secret_access_key: Refreshing state... [id=arn:aws:secretsmanager:eu-central-1:905418398257:secret:oonidevops/ooniapi_user/aws_secret_access_key-L0DQDr|terraform-20240314200140914600000006]
aws_acm_certificate_validation.ooniapi_frontend: Refreshing state... [id=2026-03-02 13:38:05.483 +0000 UTC]
module.ooniapi_oonifindings.aws_ecs_task_definition.ooniapi_service: Refreshing state... [id=ooniapi-service-oonifindings-td]
module.ooniapi_oonirun.aws_ecs_task_definition.ooniapi_service: Refreshing state... [id=ooniapi-service-oonirun-td]
module.ooniapi_reverseproxy.aws_ecs_task_definition.ooniapi_service: Refreshing state... [id=ooniapi-service-reverseproxy-td]
aws_iam_role_policy.ooniprobe_role: Refreshing state... [id=ooniapi-ecs-cluster-container-host-role:oonidevops-dev-task-role]
module.ooniapi_ooniauth.aws_ecs_task_definition.ooniapi_service: Refreshing state... [id=ooniapi-service-ooniauth-td]
data.aws_secretsmanager_secret_version.deploy_key: Reading...
aws_codestarconnections_connection.oonidevops: Refreshing state... [id=arn:aws:codestar-connections:eu-central-1:905418398257:connection/6bd492f6-c11d-43ec-92b0-24c47700d528]
data.aws_secretsmanager_secret_version.deploy_key: Read complete after 0s [id=arn:aws:secretsmanager:eu-central-1:905418398257:secret:oonidevops/deploy_key-2ebqSe|AWSCURRENT]
module.terraform_state_backend.data.aws_region.current: Reading...
module.terraform_state_backend.data.aws_region.current: Read complete after 0s [id=eu-central-1]
module.network.aws_vpc.main: Refreshing state... [id=vpc-0e382f3ad89286de9]
module.ooni_th_droplet.data.cloudinit_config.ooni_th_docker: Reading...
module.ooni_th_droplet.data.cloudinit_config.ooni_th_docker: Read complete after 0s [id=1194028725]
module.ooni_th_droplet.digitalocean_droplet.ooni_th_docker[0]: Refreshing state... [id=459912318]
module.terraform_state_backend.aws_s3_bucket.default[0]: Refreshing state... [id=oonidevops-dev-terraform-state]
module.terraform_state_backend.data.aws_iam_policy_document.bucket_policy[0]: Reading...
module.terraform_state_backend.data.aws_iam_policy_document.bucket_policy[0]: Read complete after 0s [id=2666303363]
module.terraform_state_backend.data.aws_iam_policy_document.aggregated_policy[0]: Reading...
module.terraform_state_backend.data.aws_iam_policy_document.aggregated_policy[0]: Read complete after 0s [id=2666303363]
module.terraform_state_backend.aws_dynamodb_table.with_server_side_encryption[0]: Refreshing state... [id=oonidevops-dev-terraform-state-lock]
module.fastpath_builder.aws_iam_policy.codebuild: Refreshing state... [id=arn:aws:iam::905418398257:policy/service-role/codebuild-fastpath-eu-central-1]
module.ooniapi_ooniprobe_deployer.aws_iam_policy.codebuild: Refreshing state... [id=arn:aws:iam::905418398257:policy/service-role/codebuild-ooniprobe-eu-central-1]
module.ooniapi_reverseproxy_deployer.aws_iam_policy.codebuild: Refreshing state... [id=arn:aws:iam::905418398257:policy/service-role/codebuild-reverseproxy-eu-central-1]
module.testlists_builder.aws_iam_policy.codebuild: Refreshing state... [id=arn:aws:iam::905418398257:policy/service-role/codebuild-testlists-eu-central-1]
module.ooniapi_oonimeasurements_deployer.aws_iam_policy.codebuild: Refreshing state... [id=arn:aws:iam::905418398257:policy/service-role/codebuild-oonimeasurements-eu-central-1]
module.ooniapi_oonirun_deployer.aws_iam_policy.codebuild: Refreshing state... [id=arn:aws:iam::905418398257:policy/service-role/codebuild-oonirun-eu-central-1]
module.ooniapi_oonifindings_deployer.aws_iam_policy.codebuild: Refreshing state... [id=arn:aws:iam::905418398257:policy/service-role/codebuild-oonifindings-eu-central-1]
module.ooniapi_ooniauth_deployer.aws_iam_policy.codebuild: Refreshing state... [id=arn:aws:iam::905418398257:policy/service-role/codebuild-ooniauth-eu-central-1]
module.testlists_builder.aws_iam_role.codebuild: Refreshing state... [id=codebuild-oonidkr-testlists]
module.fastpath_builder.aws_iam_role.codebuild: Refreshing state... [id=codebuild-oonidkr-fastpath]
module.ooniapi_reverseproxy_deployer.aws_iam_role.codebuild: Refreshing state... [id=codebuild-ooniapi-reverseproxy]
module.ooniapi_ooniprobe_deployer.aws_iam_role.codebuild: Refreshing state... [id=codebuild-ooniapi-ooniprobe]
module.ooniapi_oonimeasurements_deployer.aws_iam_role.codebuild: Refreshing state... [id=codebuild-ooniapi-oonimeasurements]
module.ooniapi_oonirun_deployer.aws_iam_role.codebuild: Refreshing state... [id=codebuild-ooniapi-oonirun]
module.ooniapi_oonifindings_deployer.aws_iam_role.codebuild: Refreshing state... [id=codebuild-ooniapi-oonifindings]
module.ooniapi_ooniauth_deployer.aws_iam_role.codebuild: Refreshing state... [id=codebuild-ooniapi-ooniauth]
module.testlists_builder.aws_codebuild_project.oonidkr: Refreshing state... [id=arn:aws:codebuild:eu-central-1:905418398257:project/oonidkr-testlists]
module.fastpath_builder.aws_codebuild_project.oonidkr: Refreshing state... [id=arn:aws:codebuild:eu-central-1:905418398257:project/oonidkr-fastpath]
module.ooniapi_ooniprobe_deployer.aws_codebuild_project.ooniapi: Refreshing state... [id=arn:aws:codebuild:eu-central-1:905418398257:project/ooniapi-ooniprobe]
module.ooniapi_reverseproxy_deployer.aws_codebuild_project.ooniapi: Refreshing state... [id=arn:aws:codebuild:eu-central-1:905418398257:project/ooniapi-reverseproxy]
module.ooniapi_oonimeasurements_deployer.aws_codebuild_project.ooniapi: Refreshing state... [id=arn:aws:codebuild:eu-central-1:905418398257:project/ooniapi-oonimeasurements]
module.ooniapi_oonirun_deployer.aws_codebuild_project.ooniapi: Refreshing state... [id=arn:aws:codebuild:eu-central-1:905418398257:project/ooniapi-oonirun]
module.ooniapi_oonifindings_deployer.aws_codebuild_project.ooniapi: Refreshing state... [id=arn:aws:codebuild:eu-central-1:905418398257:project/ooniapi-oonifindings]
module.ooniapi_ooniauth_deployer.aws_codebuild_project.ooniapi: Refreshing state... [id=arn:aws:codebuild:eu-central-1:905418398257:project/ooniapi-ooniauth]
module.network.aws_internet_gateway.gw: Refreshing state... [id=igw-0c080e9b235ed29d1]
module.ooni_th_droplet.aws_route53_record.ooni_th["0"]: Refreshing state... [id=Z055356431RGCLK3JXZDL_0.do.th.dev.ooni.io_A]
module.testlists_builder.aws_codepipeline.oonidkr: Refreshing state... [id=oonidkr-testlists]
module.fastpath_builder.aws_codepipeline.oonidkr: Refreshing state... [id=oonidkr-fastpath]
aws_security_group.elasticache_sg: Refreshing state... [id=sg-071f88ba24fb3d6ed]
module.ooniapi_testlists.aws_alb_target_group.ooni_ec2: Refreshing state... [id=arn:aws:elasticloadbalancing:eu-central-1:905418398257:targetgroup/ootstl20260302133704045300000001/4892b94962057fea]
module.ooniapi_testlists.aws_security_group.ec2_sg: Refreshing state... [id=sg-031fd5ff750f4c94b]
module.ooniapi_oonirun.aws_alb_target_group.ooniapi_service: Refreshing state... [id=arn:aws:elasticloadbalancing:eu-central-1:905418398257:targetgroup/OrunM-20250115122624347100000003/17e1664b99b708a5]
module.ooniapi_ooniprobe.aws_alb_target_group.ooniapi_service: Refreshing state... [id=arn:aws:elasticloadbalancing:eu-central-1:905418398257:targetgroup/OproM-20250115122624346700000001/9f9264a4e53931d3]
module.ooniapi_ooniauth.aws_alb_target_group.ooniapi_service: Refreshing state... [id=arn:aws:elasticloadbalancing:eu-central-1:905418398257:targetgroup/OautM-20250115122624347200000004/6e746a968782a49f]
module.oonitier1plus_cluster.aws_security_group.web: Refreshing state... [id=sg-07090c14e80a5def2]
module.ooni_monitoring_proxy.aws_security_group.ec2_sg: Refreshing state... [id=sg-00c4199ae6a658579]
module.ooni_monitoring_proxy.aws_alb_target_group.ooni_ec2: Refreshing state... [id=arn:aws:elasticloadbalancing:eu-central-1:905418398257:targetgroup/oomnpr20250423083217708600000002/90babad6f0c8b903]
module.ooni_jumphost.aws_alb_target_group.ooni_ec2: Refreshing state... [id=arn:aws:elasticloadbalancing:eu-central-1:905418398257:targetgroup/oojump20251216144624441200000002/52a32be88e2fcac5]
module.ooni_jumphost.aws_security_group.ec2_sg: Refreshing state... [id=sg-0ee46dd91ace739e1]
module.ooniapi_cluster.aws_security_group.web: Refreshing state... [id=sg-0187eedfe39538357]
module.ooniapi_oonifindings.aws_alb_target_group.ooniapi_service: Refreshing state... [id=arn:aws:elasticloadbalancing:eu-central-1:905418398257:targetgroup/OfinM-20250115122624350600000005/ad715c6e26dd616c]
module.ooniapi_reverseproxy.aws_alb_target_group.ooniapi_service: Refreshing state... [id=arn:aws:elasticloadbalancing:eu-central-1:905418398257:targetgroup/OrevM-20250115122624347000000002/32c2f9b4e4d3b8c4]
module.ooni_clickhouse_proxy.aws_alb_target_group.ooni_ec2: Refreshing state... [id=arn:aws:elasticloadbalancing:eu-central-1:905418398257:targetgroup/oockpr20250116192249626700000002/2e9dada4dd22c268]
module.ooniapi_oonimeasurements.aws_alb_target_group.ooniapi_service: Refreshing state... [id=arn:aws:elasticloadbalancing:eu-central-1:905418398257:targetgroup/OmeaM-20250116160254864500000001/4d88cb32eb2f381c]
module.ooni_clickhouse_proxy.aws_security_group.ec2_sg: Refreshing state... [id=sg-0903c108a44c922a5]
module.ooni_anonc.aws_alb_target_group.ooni_ec2: Refreshing state... [id=arn:aws:elasticloadbalancing:eu-central-1:905418398257:targetgroup/ooanon20251003085918842900000002/3d14866336282a65]
module.ooni_anonc.aws_security_group.ec2_sg: Refreshing state... [id=sg-063668ca077d07d17]
module.ooni_fastpath.module.ec2.aws_alb_target_group.ooni_ec2: Refreshing state... [id=arn:aws:elasticloadbalancing:eu-central-1:905418398257:targetgroup/oofstp20260423110252291900000001/63d84df4b8ea92a9]
module.ooni_fastpath.module.ec2.aws_security_group.ec2_sg: Refreshing state... [id=sg-03507eea1e0e5bb0c]
module.network.aws_route_table.private: Refreshing state... [id=rtb-011463437da96c77b]
module.network.aws_route_table.public: Refreshing state... [id=rtb-0ccb0852e6a365a95]
module.network.aws_subnet.public[1]: Refreshing state... [id=subnet-0b18966cccfc9d5ef]
module.network.aws_subnet.public[0]: Refreshing state... [id=subnet-0e7a4478be988463f]
module.network.aws_subnet.private[1]: Refreshing state... [id=subnet-0b899a7ad10406d06]
module.network.aws_subnet.private[0]: Refreshing state... [id=subnet-09314a43ec89d6331]
module.ooniapi_testlists.aws_security_group_rule.ec2_sg_egress[0]: Refreshing state... [id=sgrule-3440521332]
module.ooniapi_testlists.aws_security_group_rule.ec2_sg_egress[1]: Refreshing state... [id=sgrule-1726180651]
module.ooni_monitoring_proxy.aws_security_group_rule.ec2_sg_egress[0]: Refreshing state... [id=sgrule-4288788045]
module.ooni_monitoring_proxy.aws_security_group_rule.ec2_sg_egress[1]: Refreshing state... [id=sgrule-3806784481]
module.ooni_monitoring_proxy.aws_security_group_rule.ec2_sg_ingress[2]: Refreshing state... [id=sgrule-2383513485]
module.ooni_monitoring_proxy.aws_security_group_rule.ec2_sg_ingress[3]: Refreshing state... [id=sgrule-1109732603]
module.ooni_monitoring_proxy.aws_security_group_rule.ec2_sg_ingress[0]: Refreshing state... [id=sgrule-2756751855]
module.ooni_monitoring_proxy.aws_security_group_rule.ec2_sg_ingress[1]: Refreshing state... [id=sgrule-316337242]
module.ooni_jumphost.aws_security_group_rule.ec2_sg_egress[1]: Refreshing state... [id=sgrule-780291060]
module.ooni_jumphost.aws_security_group_rule.ec2_sg_egress[0]: Refreshing state... [id=sgrule-844844036]
module.terraform_state_backend.aws_s3_bucket_public_access_block.default[0]: Refreshing state... [id=oonidevops-dev-terraform-state]
module.terraform_state_backend.aws_s3_bucket_server_side_encryption_configuration.default[0]: Refreshing state... [id=oonidevops-dev-terraform-state]
module.terraform_state_backend.aws_s3_bucket_versioning.default[0]: Refreshing state... [id=oonidevops-dev-terraform-state]
module.ooniapi_oonirun.aws_ecs_service.ooniapi_service: Refreshing state... [id=arn:aws:ecs:eu-central-1:905418398257:service/ooniapi-ecs-cluster/ooniapi-service-oonirun]
module.ooniapi_ooniauth.aws_ecs_service.ooniapi_service: Refreshing state... [id=arn:aws:ecs:eu-central-1:905418398257:service/ooniapi-ecs-cluster/ooniapi-service-ooniauth]
module.ooni_clickhouse_proxy.aws_security_group_rule.ec2_sg_egress[0]: Refreshing state... [id=sgrule-1099643652]
module.ooni_clickhouse_proxy.aws_security_group_rule.ec2_sg_egress[1]: Refreshing state... [id=sgrule-1281654482]
module.ooni_anonc.aws_security_group_rule.ec2_sg_egress[0]: Refreshing state... [id=sgrule-3803885271]
module.ooni_anonc.aws_security_group_rule.ec2_sg_egress[1]: Refreshing state... [id=sgrule-2372809180]
module.ooniapi_oonifindings.aws_ecs_service.ooniapi_service: Refreshing state... [id=arn:aws:ecs:eu-central-1:905418398257:service/ooniapi-ecs-cluster/ooniapi-service-oonifindings]
module.ooni_fastpath.module.ec2.aws_security_group_rule.ec2_sg_egress[1]: Refreshing state... [id=sgrule-2098199074]
module.ooni_fastpath.module.ec2.aws_security_group_rule.ec2_sg_egress[0]: Refreshing state... [id=sgrule-3915082054]
module.ooniapi_reverseproxy.aws_ecs_service.ooniapi_service: Refreshing state... [id=arn:aws:ecs:eu-central-1:905418398257:service/ooniapi-ecs-cluster/ooniapi-service-reverseproxy]
module.network.aws_route_table_association.public[0]: Refreshing state... [id=rtbassoc-0dbd7fb16801ee049]
module.network.aws_route_table_association.public[1]: Refreshing state... [id=rtbassoc-08ab18165bf481054]
module.network.aws_route_table_association.private[1]: Refreshing state... [id=rtbassoc-0c9cc0f117ef15fe7]
module.network.aws_route_table_association.private[0]: Refreshing state... [id=rtbassoc-0e7933e6b804ff2c1]
module.terraform_state_backend.aws_s3_bucket_policy.default[0]: Refreshing state... [id=oonidevops-dev-terraform-state]
aws_security_group_rule.elasticache_sg_rule: Refreshing state... [id=sgrule-1589925589]
aws_elasticache_serverless_cache.ooniapi: Refreshing state... [id=ooniapi-dev-cache]
module.oonipg.aws_security_group.pg: Refreshing state... [id=sg-005ca579eb9c08cda]
module.oonitier1plus_cluster.aws_security_group.container_host: Refreshing state... [id=sg-0e74a206196727883]
module.ooniapi_cluster.aws_security_group.container_host: Refreshing state... [id=sg-0aa6a97400b619de3]
module.ooni_jumphost.aws_launch_template.ooni_ec2: Refreshing state... [id=lt-093e415469bef9855]
module.ooniapi_frontend.aws_alb.ooniapi: Refreshing state... [id=arn:aws:elasticloadbalancing:eu-central-1:905418398257:loadbalancer/app/ooni-api-frontend/4a50f3dd46584390]
module.ooni_anonc.aws_launch_template.ooni_ec2: Refreshing state... [id=lt-097d2fb5eb0bf4559]
module.ooni_monitoring_proxy.aws_launch_template.ooni_ec2: Refreshing state... [id=lt-0c9dddb576a4f71a3]
module.ooniapi_testlists.aws_launch_template.ooni_ec2: Refreshing state... [id=lt-01d0fc9e7bff14cac]
module.oonipg.aws_db_subnet_group.pg: Refreshing state... [id=ooni-tier0-postgres-dbsng]
module.ooni_clickhouse_proxy.aws_launch_template.ooni_ec2: Refreshing state... [id=lt-0855bc6373ff4c75b]
module.ooniapi_oonirun_deployer.aws_codepipeline.ooniapi: Refreshing state... [id=ooniapi-oonirun]
module.terraform_state_backend.time_sleep.wait_for_aws_s3_bucket_settings[0]: Refreshing state... [id=2024-03-10T15:06:17Z]
module.ooni_fastpath.module.ec2.aws_launch_template.ooni_ec2: Refreshing state... [id=lt-04f17f1d49bea5597]
module.oonitier1plus_cluster.aws_launch_template.container_host: Refreshing state... [id=lt-0eb432177b5a9f2aa]
module.ooniapi_cluster.aws_launch_template.container_host: Refreshing state... [id=lt-0e328a8671f870c64]
module.ooniapi_ooniauth_deployer.aws_codepipeline.ooniapi: Refreshing state... [id=ooniapi-ooniauth]
module.ooniapi_oonifindings_deployer.aws_codepipeline.ooniapi: Refreshing state... [id=ooniapi-oonifindings]
module.ooni_jumphost.aws_instance.ooni_ec2: Refreshing state... [id=i-0ab8df111ab0fa5a3]
module.terraform_state_backend.aws_s3_bucket_ownership_controls.default[0]: Refreshing state... [id=oonidevops-dev-terraform-state]
module.ooni_monitoring_proxy.aws_instance.ooni_ec2: Refreshing state... [id=i-067b337ada2d9cc00]
module.ooni_anonc.aws_instance.ooni_ec2: Refreshing state... [id=i-058b0fd97a772f7e1]
module.ooniapi_frontend.aws_alb_listener.ooniapi_listener_https: Refreshing state... [id=arn:aws:elasticloadbalancing:eu-central-1:905418398257:listener/app/ooni-api-frontend/4a50f3dd46584390/9ef650e256f41d45]
module.ooniapi_frontend.aws_alb_listener.ooniapi_listener_http: Refreshing state... [id=arn:aws:elasticloadbalancing:eu-central-1:905418398257:listener/app/ooni-api-frontend/4a50f3dd46584390/664a34cfb30f72e8]
module.ooniapi_testlists.aws_instance.ooni_ec2: Refreshing state... [id=i-07217491c5a2f9d77]
module.ooni_clickhouse_proxy.aws_instance.ooni_ec2: Refreshing state... [id=i-0757310827bc642fa]
module.ooni_fastpath.module.ec2.aws_instance.ooni_ec2: Refreshing state... [id=i-0eeb4d9320fc9dd3a]
module.ooniapi_reverseproxy_deployer.aws_codepipeline.ooniapi: Refreshing state... [id=ooniapi-reverseproxy]
module.oonitier1plus_cluster.aws_autoscaling_group.container_host: Refreshing state... [id=oonitier1plus-ecs-cluster20251022145227179100000007]
module.ooniapi_cluster.aws_autoscaling_group.container_host: Refreshing state... [id=ooniapi-ecs-cluster20240310192644083800000003]
aws_route53_record.ooniapi_frontend_main: Refreshing state... [id=Z055356431RGCLK3JXZDL_api.dev.ooni.io_A]
aws_route53_record.ooniapi_frontend_alt["8.th.dev.ooni.io"]: Refreshing state... [id=Z055356431RGCLK3JXZDL_8.th.dev.ooni.io_A]
aws_route53_record.ooniapi_frontend_alt["oonirun.dev.ooni.io"]: Refreshing state... [id=Z055356431RGCLK3JXZDL_oonirun.dev.ooni.io_A]
aws_route53_record.ooniapi_frontend_alt["ooniprobe.dev.ooni.io"]: Refreshing state... [id=Z055356431RGCLK3JXZDL_ooniprobe.dev.ooni.io_A]
aws_route53_record.ooniapi_frontend_alt["ooniauth.dev.ooni.io"]: Refreshing state... [id=Z055356431RGCLK3JXZDL_ooniauth.dev.ooni.io_A]
aws_route53_record.ooniapi_frontend_alt["testlists.dev.ooni.io"]: Refreshing state... [id=Z055356431RGCLK3JXZDL_testlists.dev.ooni.io_A]
aws_route53_record.ooniapi_frontend_alt["oonimeasurements.dev.ooni.io"]: Refreshing state... [id=Z055356431RGCLK3JXZDL_oonimeasurements.dev.ooni.io_A]
module.oonipg.aws_db_instance.pg: Refreshing state... [id=db-27N7Q6XIBNASFCOXN4N7C762L4]
module.ooniapi_frontend.aws_lb_listener_rule.ooniapi_oonirun_rule_host: Refreshing state... [id=arn:aws:elasticloadbalancing:eu-central-1:905418398257:listener-rule/app/ooni-api-frontend/4a50f3dd46584390/9ef650e256f41d45/9af03e886f8803f2]
module.ooniapi_frontend.aws_lb_listener_rule.ooniapi_oonifindings_rule_host: Refreshing state... [id=arn:aws:elasticloadbalancing:eu-central-1:905418398257:listener-rule/app/ooni-api-frontend/4a50f3dd46584390/9ef650e256f41d45/54cda6e694a0103f]
module.ooniapi_frontend.aws_lb_listener_rule.ooniapi_oonimeasurements_rule_2[0]: Refreshing state... [id=arn:aws:elasticloadbalancing:eu-central-1:905418398257:listener-rule/app/ooni-api-frontend/4a50f3dd46584390/9ef650e256f41d45/e6dbe09be108b001]
module.ooniapi_frontend.aws_lb_listener_rule.ooniapi_testlists_rule[0]: Refreshing state... [id=arn:aws:elasticloadbalancing:eu-central-1:905418398257:listener-rule/app/ooni-api-frontend/4a50f3dd46584390/9ef650e256f41d45/2864c2a72dfd4018]
module.ooniapi_frontend.aws_lb_listener_rule.ooniapi_oonimeasurements_rule_host[0]: Refreshing state... [id=arn:aws:elasticloadbalancing:eu-central-1:905418398257:listener-rule/app/ooni-api-frontend/4a50f3dd46584390/9ef650e256f41d45/f3d75d5d93fd6903]
module.ooniapi_frontend.aws_alb_listener_rule.ooniapi_th: Refreshing state... [id=arn:aws:elasticloadbalancing:eu-central-1:905418398257:listener-rule/app/ooni-api-frontend/4a50f3dd46584390/9ef650e256f41d45/775cd6d0dc062fd3]
module.ooniapi_frontend.aws_lb_listener_rule.ooniapi_ooniprobe_rule_2: Refreshing state... [id=arn:aws:elasticloadbalancing:eu-central-1:905418398257:listener-rule/app/ooni-api-frontend/4a50f3dd46584390/9ef650e256f41d45/5f2394ffa8b71f98]
module.ooniapi_frontend.aws_lb_listener_rule.ooniapi_ooniauth_rule: Refreshing state... [id=arn:aws:elasticloadbalancing:eu-central-1:905418398257:listener-rule/app/ooni-api-frontend/4a50f3dd46584390/9ef650e256f41d45/178511e1b6ae89c5]
module.ooniapi_frontend.aws_lb_listener_rule.ooniapi_ooniprobe_rule: Refreshing state... [id=arn:aws:elasticloadbalancing:eu-central-1:905418398257:listener-rule/app/ooni-api-frontend/4a50f3dd46584390/9ef650e256f41d45/82069bb29bca6af1]
module.ooniapi_frontend.aws_lb_listener_rule.ooniapi_oonifindings_rule: Refreshing state... [id=arn:aws:elasticloadbalancing:eu-central-1:905418398257:listener-rule/app/ooni-api-frontend/4a50f3dd46584390/9ef650e256f41d45/36d49e835c0b81c5]
module.ooniapi_frontend.aws_lb_listener_rule.ooniapi_ooniprobe_rule_3: Refreshing state... [id=arn:aws:elasticloadbalancing:eu-central-1:905418398257:listener-rule/app/ooni-api-frontend/4a50f3dd46584390/9ef650e256f41d45/e57ed4ddc7991afe]
module.ooniapi_frontend.aws_lb_listener_rule.ooniapi_ooniauth_rule_host: Refreshing state... [id=arn:aws:elasticloadbalancing:eu-central-1:905418398257:listener-rule/app/ooni-api-frontend/4a50f3dd46584390/9ef650e256f41d45/f4bf91203c7ca76e]
module.ooniapi_frontend.aws_lb_listener_rule.ooniapi_oonimeasurements_rule_3: Refreshing state... [id=arn:aws:elasticloadbalancing:eu-central-1:905418398257:listener-rule/app/ooni-api-frontend/4a50f3dd46584390/9ef650e256f41d45/48b97ed405c9c6cc]
module.ooniapi_frontend.aws_lb_listener_rule.ooniapi_ooniprobe_rule_4: Refreshing state... [id=arn:aws:elasticloadbalancing:eu-central-1:905418398257:listener-rule/app/ooni-api-frontend/4a50f3dd46584390/9ef650e256f41d45/6026e8ac5ba962ec]
module.ooniapi_frontend.aws_lb_listener_rule.ooniapi_ooniprobe_rule_host: Refreshing state... [id=arn:aws:elasticloadbalancing:eu-central-1:905418398257:listener-rule/app/ooni-api-frontend/4a50f3dd46584390/9ef650e256f41d45/583471b0bdc1c388]
module.ooniapi_frontend.aws_lb_listener_rule.ooniapi_oonirun_rule: Refreshing state... [id=arn:aws:elasticloadbalancing:eu-central-1:905418398257:listener-rule/app/ooni-api-frontend/4a50f3dd46584390/9ef650e256f41d45/cc29701b6ed6aa2e]
module.ooniapi_frontend.aws_lb_listener_rule.ooniapi_oonimeasurements_rule_1[0]: Refreshing state... [id=arn:aws:elasticloadbalancing:eu-central-1:905418398257:listener-rule/app/ooni-api-frontend/4a50f3dd46584390/9ef650e256f41d45/1cf3d6a7a694eec9]
module.oonitier1plus_cluster.aws_ecs_capacity_provider.capacity_provider: Refreshing state... [id=arn:aws:ecs:eu-central-1:905418398257:capacity-provider/oonitier1plus-ecs-cluster-capacity-provider]
module.ooniapi_cluster.aws_ecs_capacity_provider.capacity_provider: Refreshing state... [id=arn:aws:ecs:eu-central-1:905418398257:capacity-provider/ooniapi-ecs-cluster-capacity-provider]
aws_route53_record.postgres_dns: Refreshing state... [id=Z091407123AEJO90Z3H6D_postgres.dev.ooni.nu_CNAME]
data.aws_secretsmanager_secret_version.pg_login: Reading...
module.ooni_anonc.aws_lb_target_group_attachment.oonibackend_proxy: Refreshing state... [id=arn:aws:elasticloadbalancing:eu-central-1:905418398257:targetgroup/ooanon20251003085918842900000002/3d14866336282a65-20251003085941554000000006]
module.ooni_monitoring_proxy.aws_lb_target_group_attachment.oonibackend_proxy: Refreshing state... [id=arn:aws:elasticloadbalancing:eu-central-1:905418398257:targetgroup/oomnpr20250423083217708600000002/90babad6f0c8b903-20250423083239704200000006]
aws_route53_record.anonc_alias: Refreshing state... [id=Z055356431RGCLK3JXZDL_anonc.dev.ooni.io_CNAME]
aws_route53_record.monitoring_proxy_alias: Refreshing state... [id=Z055356431RGCLK3JXZDL_monitoringproxy.dev.ooni.io_CNAME]
module.oonitier1plus_cluster.aws_ecs_cluster_capacity_providers.cluster_capacity_providers: Refreshing state... [id=oonitier1plus-ecs-cluster]
module.ooniapi_testlists.aws_security_group_rule.ec2_sg_ingress[3]: Refreshing state... [id=sgrule-2701186690]
data.aws_secretsmanager_secret_version.pg_login: Read complete after 0s [id=arn:aws:secretsmanager:eu-central-1:905418398257:secret:rds!db-5fe27151-3a37-44e0-a5bd-3517363fa2e8-BDI0KI|AWSCURRENT]
module.ooniapi_testlists.aws_security_group_rule.ec2_sg_ingress[0]: Refreshing state... [id=sgrule-2460724300]
module.ooniapi_testlists.aws_security_group_rule.ec2_sg_ingress[1]: Refreshing state... [id=sgrule-606204409]
module.ooniapi_testlists.aws_security_group_rule.ec2_sg_ingress[2]: Refreshing state... [id=sgrule-14456781]
module.ooni_anonc.aws_security_group_rule.ec2_sg_ingress[1]: Refreshing state... [id=sgrule-164247457]
module.ooni_anonc.aws_security_group_rule.ec2_sg_ingress[3]: Refreshing state... [id=sgrule-2749190535]
module.ooni_anonc.aws_security_group_rule.ec2_sg_ingress[4]: Refreshing state... [id=sgrule-3453785268]
module.ooni_anonc.aws_security_group_rule.ec2_sg_ingress[2]: Refreshing state... [id=sgrule-2843886495]
module.ooni_anonc.aws_security_group_rule.ec2_sg_ingress[0]: Refreshing state... [id=sgrule-3208669716]
module.ooni_jumphost.aws_lb_target_group_attachment.oonibackend_proxy: Refreshing state... [id=arn:aws:elasticloadbalancing:eu-central-1:905418398257:targetgroup/oojump20251216144624441200000002/52a32be88e2fcac5-20251216144651363600000006]
module.ooni_jumphost.aws_security_group_rule.ec2_sg_ingress[1]: Refreshing state... [id=sgrule-1099580958]
module.ooni_jumphost.aws_security_group_rule.ec2_sg_ingress[2]: Refreshing state... [id=sgrule-1528835277]
module.ooni_jumphost.aws_security_group_rule.ec2_sg_ingress[0]: Refreshing state... [id=sgrule-4143979435]
module.ooniapi_cluster.aws_ecs_cluster_capacity_providers.cluster_capacity_providers: Refreshing state... [id=ooniapi-ecs-cluster]
aws_secretsmanager_secret_version.oonipg_url: Refreshing state... [id=arn:aws:secretsmanager:eu-central-1:905418398257:secret:oonidevops/ooni-tier0-postgres/postgresql_url-w62CTZ|terraform-20260723112126792700000001]
module.ooniapi_testlists.aws_lb_target_group_attachment.oonibackend_proxy: Refreshing state... [id=arn:aws:elasticloadbalancing:eu-central-1:905418398257:targetgroup/ootstl20260302133704045300000001/4892b94962057fea-20260302133730682200000008]
module.ooni_clickhouse_proxy.aws_lb_target_group_attachment.oonibackend_proxy: Refreshing state... [id=arn:aws:elasticloadbalancing:eu-central-1:905418398257:targetgroup/oockpr20250116192249626700000002/2e9dada4dd22c268-20260227131827648300000002]
module.ooni_fastpath.module.ec2.aws_lb_target_group_attachment.oonibackend_proxy: Refreshing state... [id=arn:aws:elasticloadbalancing:eu-central-1:905418398257:targetgroup/oofstp20260423110252291900000001/63d84df4b8ea92a9-20260423110318998700000007]
aws_route53_record.jumphost_alias: Refreshing state... [id=Z055356431RGCLK3JXZDL_jumphost.dev.ooni.io_CNAME]
aws_route53_record.testlists_alias: Refreshing state... [id=Z055356431RGCLK3JXZDL_testlist-ec2.dev.ooni.io_CNAME]
aws_route53_record.clickhouse_proxy_alias: Refreshing state... [id=Z055356431RGCLK3JXZDL_clickhouseproxy.dev.ooni.io_CNAME]
module.ooni_fastpath.aws_route53_record.fastpath_alias: Refreshing state... [id=Z055356431RGCLK3JXZDL_fastpath.dev.ooni.io_CNAME]
module.ooni_fastpath.module.ec2.aws_security_group_rule.ec2_sg_ingress[1]: Refreshing state... [id=sgrule-2656646093]
module.ooni_fastpath.module.ec2.aws_security_group_rule.ec2_sg_ingress[2]: Refreshing state... [id=sgrule-3549062282]
module.ooni_fastpath.module.ec2.aws_security_group_rule.ec2_sg_ingress[0]: Refreshing state... [id=sgrule-4279797320]
module.ooni_fastpath.module.ec2.aws_security_group_rule.ec2_sg_ingress[4]: Refreshing state... [id=sgrule-858198288]
module.ooni_fastpath.module.ec2.aws_security_group_rule.ec2_sg_ingress[3]: Refreshing state... [id=sgrule-3011210381]
module.ooni_clickhouse_proxy.aws_security_group_rule.ec2_sg_ingress[0]: Refreshing state... [id=sgrule-1921217342]
module.ooni_clickhouse_proxy.aws_security_group_rule.ec2_sg_ingress[2]: Refreshing state... [id=sgrule-522211670]
module.ooni_clickhouse_proxy.aws_security_group_rule.ec2_sg_ingress[1]: Refreshing state... [id=sgrule-3288936075]
module.ooni_clickhouse_proxy.aws_security_group_rule.ec2_sg_ingress[3]: Refreshing state... [id=sgrule-3953292375]
module.ooni_clickhouse_proxy.aws_security_group_rule.ec2_sg_ingress[4]: Refreshing state... [id=sgrule-3520426823]
nomad_job.oonimeasurements: Refreshing state... [id=oonimeasurements]
module.ooniapi_ooniprobe.aws_ecs_task_definition.ooniapi_service: Refreshing state... [id=ooniapi-service-ooniprobe-td]
module.ooniapi_ooniprobe.aws_ecs_service.ooniapi_service: Refreshing state... [id=arn:aws:ecs:eu-central-1:905418398257:service/ooniapi-ecs-cluster/ooniapi-service-ooniprobe]
module.ooniapi_ooniprobe_deployer.aws_codepipeline.ooniapi: Refreshing state... [id=ooniapi-ooniprobe]

Planning failed. Terraform encountered an error while generating this plan.


Warning: Argument is deprecated

  with aws_s3_bucket.anoncred_manifests,
  on main.tf line 285, in resource "aws_s3_bucket" "anoncred_manifests":
 285: resource "aws_s3_bucket" "anoncred_manifests" {

Use the aws_s3_bucket_versioning resource instead

(and 5 more similar warnings elsewhere)

Warning: Deprecated value used

  on ../../modules/ecs_cluster/outputs.tf line 22, in output "container_host_role":
  22:   value = aws_iam_role.container_host

  The deprecation originates from module.ooniapi_cluster.aws_iam_role.container_host.inline_policy

Deprecated resource block "inline_policy" used. Refer to the provider
documentation for details.

(and 7 more similar warnings elsewhere)

Warning: Available Write-only Attribute Alternative

  with module.ooni_monitoring.aws_ssm_parameter.ooni_monitoring_access_key,
  on ../../modules/ooni_monitoring/main.tf line 47, in resource "aws_ssm_parameter" "ooni_monitoring_access_key":
  47:   value = aws_iam_access_key.ooni_monitoring.id

The attribute value has a write-only alternative value_wo available. Use the
write-only alternative of the attribute when possible.

(and one more similar warning elsewhere)


Pusher	@LDiazN
Action	pull_request
Environment	dev
Workflow	.github/workflows/check_terraform.yml
Last updated	Mon, 27 Jul 2026 15:27:08 GMT

LDiazN added 30 commits July 16, 2026 16:41

Add nomad installation

c9e061f

Add nomad config file

430f31f

Install docker in nomad server

c3bc013

Add relevant tags to every service

1520461

Add dehydrated and nginx for https

be8c6b8

Open nomad ports

9d367ce

Add nomad webui port

5401687

Add tls to nomad webui

9b2ba52

Add basic auth to nomad

9b9ccdf

Use password in dev env

5b6093d

deactivate internal tls (nginx handles tls)

212bb7f

Add nomad provider to terraform

8c5939b

Add nomad to terraform

5eaacd8

remove podman

dee04ac

Add job directory with oonimeasurements job

ba74b1b

restore podman

61feb22

Add nomad nginx configs

284fe43

Fix nginx config

125406e

Final settings to support exec on browser

d6b054e

Add nginx for api

dfd1e39

Remove redundant prometheus file

fe4a270

Add anonc to clickhouse proxy rules

9e3f3d7

Change clickhouse proxy url

7a1857b

Add health checking

e72f4f6

Add plugin dir configuration to nomad

38292a1

Add nomad tags

8bb024b

Start podman service; setup podman driver

3c0aad0

Change driver from docker to podman

4e39209

Add docker prefix to image name

28c3ff4

Create podman user for rootless containers

622cd5d

LDiazN added 7 commits July 23, 2026 14:07

Setup service to run as rootless

9a12d22

Fix bug with uid space

49e91e5

Add nomad tags

be19062

Remove user config in job

452df97

Add template for generic jobs

4189092

Add valkey job

518dc3a

Add canary deployment to oonimeasurements

6301f73

LDiazN self-assigned this Jul 27, 2026

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Service deployment on Nomad POC - #461

Service deployment on Nomad POC#461
LDiazN wants to merge 37 commits into
mainfrom
nomad-experiment

LDiazN commented Jul 27, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Jul 27, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Jul 27, 2026 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

LDiazN commented Jul 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Background

This PR

Highlights

Cons

Uh oh!

github-actions Bot commented Jul 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Ansible Run Output 🤖

Ansible Playbook Recap 🔍

Ansible playbook output 📖success

Uh oh!

github-actions Bot commented Jul 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Terraform Run Output 🤖

Format and Style 🖌failure

Initialization ⚙️success

Validation 🤖success

Plan 📖success

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

LDiazN commented Jul 27, 2026 •

edited

Loading

github-actions Bot commented Jul 27, 2026 •

edited

Loading

Ansible playbook output 📖`success`

github-actions Bot commented Jul 27, 2026 •

edited

Loading

Format and Style 🖌`failure`

Initialization ⚙️`success`

Validation 🤖`success`

Plan 📖`success`