feat(semconv): update to v1.25 in a non-breaking way (#1651)

* use v1.10-compat semconvgen to write deprecation notices

* update semconv rake targets and templates for v1.26

* pull from the new semconv repo

* update jinja2 template for the new structure of convention data

* stable semconv names will appear in OpenTelemetry::SemanticConventions
  while all semconv names (experimental, deprecated, stable, etc) appear
  in OpenTelemetry::SemanticCandidates
    - This maps to what other langs (Java, Python, JS) are putting into
      "Incubating" modules. We opted for "candidates" because
      OpenTelemetry::SemanticConventions::Incubating::... was getting
      too long and OpenTelemetry::SemanticIncubating didn't make much
      sense.

* generate a module per root semconv namespace to make navigating docs
  more focused and to decrease the number of constants loaded when
  using more precise requires in downstream code

* generates only attribute name constants and to an attributes subfolder
  to prepare for metric name constants to be generated to metrics
  subfolders, also for decreasing the scope of what gets loaded when
  downstream instrumentation requires these constants

Co-authored-by: Kayla Reopelle (she/her) <[email protected]>

* the code generated

* add ability to exclude specific attrs from code gen

Because converting to SCREAMING_SNAKE_CASE cannot distinguish
between ...

    screaming.snake_case

... and ...

    screaming.snake.case

So, like other languages, ignore the deprecated messaging.client_id in
favor of its replacement and let backends deal with the changed attr
name.

* reference the full name in the deprecation notice toward stable

* test that constants are available in candidates

    277 runs, 500 assertions, 26 failures, 0 errors, 0 skips

Apparently there are some kinks to work out.

* generate v1.25 semconv

There's naming hijinks in the db.* namespace I would like to avoid in
this first effort to bring semconv up-to-recent-date.

* add/expand doc comments on top-level modules

Explain the auto-generated situation.

* correct deprecation notices for several old names

The v1.10 names whose comments were updated in this commit were renamed
or removed in version 1.11-1.25. Made note of their replacements or that
their use should be dropped.

Removed test for one-to-one matching of 1.10 constants to 1.25 "all"
constants. That will never pass without updating the source semconv YAML
to add back the missing names with dep notices. I believe our dep notes
will suffice and we won't auto-generate Resource or Trace again.

* gitignore semconv repo clone; rake improvements

* Move to a tmp/ dir and ignore that.

* Update the git commands to be able to reuse one directory for
  the repo working copy, just update what's needed for a different
  tag between iterations. (Generally only an improvement for dev envs.)

* Use sh() for shell commands so that tasks fail if their commands fail.

* Appease Rubocop.

* typo/grammar corrections

Co-authored-by: Kayla Reopelle <[email protected]>

* adds kramdown to dev dependencies

Yard's default use of the rdoc markdown provider was choking on links
containing codeblocks, for example: [`ENV`](link/to/somewhere). These
style links are pretty popular in the text details provided with semconv
attributes.

Let's switch to kramdown which is a markdown parser and
renderer that:

* can handle inline code blocks inside a link anchor
* is pure Ruby (no C extensions) so works with JRuby

* reorg module namespaces

* out with Candidates, we're using Incubating like most other SIGs
* shorten the namespace to SemConv
* do not provide an all-in-one rollup require for all namespaces;
  downstream users must directly require the namespaces that they use
* attributes that are templates (prefixes, really, for the moment) are
  generated as lambdas with _LAMBDA appended to their name; users must
  call that lambda with a key for the template to return the attribute
  name string

* semconv 1.26 with weaver

With multiple template entries in weaver.yaml, code generation for
stable and incubating can be performed in one invocation of the docker
image. Rakefile targets trimmed down in light of that.

* generate separate attrs and metrics files

Each root namespace gets separate attributes and metrics files.

Include examples of possible values for attributes.

Include a usage example of the template attributes as lambdas.

Fix links in doc comments referencing stable constants.

Co-authored-by: Hannah Ramadan <[email protected]>
Co-authored-by: Kayla Reopelle <[email protected]>

* generate rollup/barrel requires for each root namespace

* extract clean of generated code into separate task

* appease Rubocop

* Layout/HeredocIndentation: Use 2 spaces for indentation in a
  heredoc.
* Style/StringLiterals: Prefer single-quoted strings when you don't
  need string interpolation or special symbols.

* Style/StringConcatenation: Prefer string interpolation to string
  concatenation.
  - ... except we don't want *interpolation*; we want to join pathname
  components with the correct separator for the operating system. So
  we'll use Pathname.join explicitly instead of Pathname's :+ operator
  so Rubocop doesn't think we're concatonating strings.

* appease Rubocop

* Style/RedundantCurrentDirectoryInPath: Remove the redundant
  current directory path
* Style/RedundantFileExtensionInRequire: Redundant .rb file
  extension detected.

* correct yard-doc refs to new short namespaces

* Update Rakefile, tests, Gemfile

* Bump constants to 1.36.0

* appease Rubocop

Bundler/OrderedGems: Gems should be sorted in an alphabetical order within their section of the Gemfile.

Minitest/EmptyLineBeforeAssertionMethods: Add empty line before assertion.

Co-authored-by: Kayla Reopelle <[email protected]>

---------

Co-authored-by: Kayla Reopelle (she/her) <[email protected]>
Co-authored-by: Hannah Ramadan <[email protected]>
Co-authored-by: Kayla Reopelle <[email protected]>
Co-authored-by: Kayla Reopelle <[email protected]>

Author: 3 people

semantic_conventions/.gitignore

@@ -0,0 +1 @@
+tmp/

semantic_conventions/.rubocop.yml

@@ -26,6 +26,8 @@ Metrics/MethodLength:
 Metrics/ModuleLength:
   Enabled: false
 
+Naming/ClassAndModuleCamelCase: # because the source for generating module names isn't Rubyish enough
+  Enabled: false
 Naming/FileName:
   Exclude:
     - lib/opentelemetry-semantic_conventions.rb

semantic_conventions/.yardopts

@@ -1,9 +1,9 @@
 --no-private
 --title=OpenTelemetry Semantic Conventions
 --markup=markdown
+--markup-provider=kramdown
 --main=README.md
-./lib/opentelemetry/semantic_conventions/**/*.rb
-./lib/opentelemetry/semantic_conventions.rb
+./lib/opentelemetry/**/*.rb
 -
 README.md
 CHANGELOG.md

semantic_conventions/Gemfile

@@ -11,6 +11,7 @@ gemspec
 eval_gemfile '../contrib/Gemfile.shared'
 
 group :development, :test do
+  gem 'mutex_m' if RUBY_VERSION >= '3.4'
   gem 'opentelemetry-api', path: '../api'
   gem 'pry'
 end

semantic_conventions/Rakefile

@@ -1,16 +1,29 @@
 # frozen_string_literal: true
 
-# Copyright The OpenTelemetry Authors
-#
-# SPDX-License-Identifier: Apache-2.0
+COPYRIGHT_NOTICE = <<~COPYRIGHT_NOTICE
+  # Copyright The OpenTelemetry Authors
+  #
+  # Licensed under the Apache License, Version 2.0 (the "License");
+  # you may not use this file except in compliance with the License.
+  # You may obtain a copy of the License at
+  #
+  #     http://www.apache.org/licenses/LICENSE-2.0
+  #
+  # Unless required by applicable law or agreed to in writing, software
+  # distributed under the License is distributed on an "AS IS" BASIS,
+  # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+  # See the License for the specific language governing permissions and
+  # limitations under the License.
+  #
+  # SPDX-License-Identifier: Apache-2.0
+COPYRIGHT_NOTICE
 
 require 'bundler/gem_tasks'
 require 'rake/testtask'
 require 'yard'
 require 'rubocop/rake_task'
 require 'tmpdir'
-
-SPEC_VERSION = '1.10.0'
+require 'pathname'
 
 RuboCop::RakeTask.new
 
@@ -35,34 +48,85 @@ default_tasks =
 task default: default_tasks
 
 desc 'update semantic conventions'
-task :generate do
-  cwd = Dir.pwd
-
-  Dir.mktmpdir('opentelemetry-specification', Dir.pwd) do |tmpdir|
-    `git clone https://github.com/open-telemetry/opentelemetry-specification.git #{tmpdir}`
-    Dir.chdir(tmpdir) do
-      `git fetch`
-      `git checkout "v#{SPEC_VERSION}"`
-    end
+task generate: %i[update_gem_version generate_semconv generate_require_rollups rubocop_autocorrect]
 
-    %w[trace resource].each do |kind|
-      cmd = %W[
-        docker run --rm
-        -v "#{tmpdir}/semantic_conventions/#{kind}":/source
-        -v "#{cwd}/templates":/templates
-        -v "#{cwd}/lib":/output
-        otel/semconvgen:0.11.1
-        -f /source code
-        --template /templates/semantic_conventions.j2
-        --output /output/opentelemetry/semantic_conventions/#{kind}.rb
-        -Dmodule=#{kind[0].upcase}#{kind[1..]}
-      ]
-
-      puts "Running: #{cmd.join(' ')}"
-      `#{cmd.join(' ')}`
-    end
+SPEC_VERSION = '1.36.0'
+OTEL_WEAVER_VERSION = 'v0.17.1'
+semconv_source_dir = Pathname.new('./tmp/semconvrepo')
+semconv_output_dir = Pathname.new('./lib/opentelemetry/semconv')
+
+desc 'Run safe autocorrect on Rubocop to fix spacing issues'
+task :rubocop_autocorrect do
+  sh('bundle exec rubocop -a')
+end
+
+directory semconv_source_dir do
+  puts "\n+++ Cloning semantic conventions repository\n"
+  sh "git clone --tags --depth=1 --branch v#{SPEC_VERSION} https://github.com/open-telemetry/semantic-conventions.git #{semconv_source_dir}"
+end
+
+task check_out_semconv_version: [semconv_source_dir] do
+  puts "\n+++ Checking out semantic conventions version #{SPEC_VERSION}\n"
+  Dir.chdir(semconv_source_dir) do
+    sh "git fetch --depth 1 origin tag v#{SPEC_VERSION}"
+    sh "git checkout 'v#{SPEC_VERSION}'"
   end
+end
+
+desc 'Remove previously generated semantic conventions code.'
+task :clean_generated_code do
+  puts "\n+++ Removing previously generated semantic conventions code.\n"
+  semconv_output_dir
+    .glob('**/*.rb')
+    .then { |files| FileUtils.rm(files) }
+end
+
+task generate_semconv: %i[check_out_semconv_version clean_generated_code] do
+  puts "\n+++ Generating semantic conventions code.\n"
+  sh <<~DOCKER_COMMAND
+    docker run --rm \
+      -v "#{semconv_source_dir}/model":/source \
+      -v ./templates:/templates \
+      -v ./:/output \
+      otel/weaver:#{OTEL_WEAVER_VERSION} \
+      registry generate \
+      --registry=/source \
+      --templates=/templates \
+      ruby \
+      /output/#{semconv_output_dir}
+  DOCKER_COMMAND
+end
+
+task generate_require_rollups: %i[generate_semconv] do
+  puts "\n+++ Generating rollup/barrel files for semconv namespaces.\n"
+
+  Rake::FileList[  # find all the generated semconv files
+    semconv_output_dir.join('**', 'attributes.rb'),
+    semconv_output_dir.join('**', 'metrics.rb')
+  ]
+    .pathmap('%d') # turn the list into the parent directories (root_namespaces)
+    .uniq          # remove duplicates
+    .each do |namespace_dir| # in each directory, write out a file to require the generated signal names
+      directory = Pathname.new(namespace_dir)
+      rollup_filename = directory.dirname + "#{directory.basename}.rb"
+      File.open(rollup_filename, 'w') do |rollup|
+        rollup << <<~FILE_HEADER
+          # frozen_string_literal: true
+
+          #{COPYRIGHT_NOTICE}
+          # This file was autogenerated. Do not edit it by hand.
+
+        FILE_HEADER
+        rollup << "require_relative '#{directory.basename}/attributes'\n" if File.exist?(directory.join('attributes.rb'))
+        rollup << "require_relative '#{directory.basename}/metrics'\n" if File.exist?(directory.join('metrics.rb'))
+      end
+      puts "✅ Generated file \"#{rollup_filename}\""
+    end
+end
 
-  `sed -i.bak "s/VERSION = '.*'/VERSION = '#{SPEC_VERSION}'/g" lib/opentelemetry/semantic_conventions/version.rb`
-  `rm lib/opentelemetry/semantic_conventions/version.rb.bak`
+desc 'Bump the semantic_conventions gem version to match the spec'
+task :update_gem_version do
+  puts "\n+++ Updating gem version to #{SPEC_VERSION}\n"
+  sh %(sed -i.bak "s/VERSION = '.*'/VERSION = '#{SPEC_VERSION}'/g" lib/opentelemetry/semantic_conventions/version.rb)
+  sh 'rm lib/opentelemetry/semantic_conventions/version.rb.bak'
 end

semantic_conventions/lib/opentelemetry/semantic_conventions.rb

@@ -5,10 +5,22 @@
 # SPDX-License-Identifier: Apache-2.0
 
 module OpenTelemetry
-  # Auto-generated semantic convention constants.
+  # OpenTelemetry semantic convention attribute names as constants.
+  # These are auto-generated from source YAML in {https://github.com/open-telemetry/semantic-conventions the semantic-conventions repository}.
+  #
+  # Except for the {Resource} and {Trace} modules, constants in this namespace have been declared stable by the OpenTelemetry semantic conventions working group.
+  #
+  # @note The constants here ought to remain within major versions of this library.
+  #   However, there risk with auto-generated code.
+  #   The maintainers try to prevent constants disappearing, but we cannot gaurantee this.
+  #   We strongly recommend that any constants you use in your code are exercised in your test suite to catch missing constants before release or production runtime.
   module SemanticConventions
   end
 end
 
+# TODO: test to make sure the trace and resource constants are present in SemConv::Incubating
+# TODO: test to make sure the SemConv (stable) constants are all still present in the SemConv::Incubating constants
+# TODO: remove these convenience requires in the next major version
 require_relative 'semantic_conventions/trace'
 require_relative 'semantic_conventions/resource'
+# TODO: we're not going to add any more convenience requires here; require directly what you use