Skip to content

Update schema-publish.sh #4376

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking β€œSign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Feb 25, 2025
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion package.json
Original file line number Diff line number Diff line change
@@ -14,7 +14,7 @@
"license": "Apache-2.0",
"scripts": {
"build": "bash ./scripts/md2html/build.sh",
"build-src": "npm run validate-markdown && bash ./scripts/md2html/build.sh src",
"build-src": "npm run validate-markdown && bash ./scripts/md2html/build.sh src && bash ./scripts/schema-publish.sh src",
"test": "c8 --100 vitest --watch=false && bash scripts/schema-test-coverage.sh",
"format-markdown": "bash ./scripts/format-markdown.sh ./src/oas.md",
"validate-markdown": "npx mdv src/oas.md && npx markdownlint-cli src/oas.md"
82 changes: 55 additions & 27 deletions scripts/schema-publish.sh
Original file line number Diff line number Diff line change
@@ -6,15 +6,61 @@

schemaDir="src/schemas/validation"
branch=$(git branch --show-current)
version=${branch:1:3}
echo === Building schemas into ./deploy/oas/$version


if [ -z "$1" ]; then
if [[ $branch =~ ^v([0-9]+\.[0-9]+)-dev$ ]]; then
deploydir="./deploy/oas/${BASH_REMATCH[1]}"
else
echo "Unable to determine version from branch name; should be vMAJOR.MINOR.PATCH-dev"
exit 1
fi
elif [ $1 = "src" ]; then
deploydir="./deploy-preview"
else
echo "Unrecognized argument"
exit 1
fi

# create the date-stamped schemas
publish_schema() {
local schema="$1"
local date="$2"
local sedCmd="$3"

local base=$(basename $schema '.yaml')
local target=$deploydir/$base/$date

mkdir -p $deploydir/$base
# replace the WORK-IN-PROGRESS placeholders
sed -e $sedCmd $schemaDir/$schema > $target.yaml

node scripts/yaml2json/yaml2json.js "$target.yaml"
rm "$target.yaml"
mv "$target.json" "$target"

# Find the jekyll lander markdown file for this iteration.
local jekyllLander=$(find "$deploydir/$base" -maxdepth 1 -name "*.md")

# Move the jekyll lander markdown for this iteration to the deploy destination.
# The lander files only exist in the gh-pages branch.
if [ ! -z "$jekyllLander" ]; then
mv $jekyllLander $target.md
echo " * $newestCommitDate: $schema & jekyll lander $(basename $jekyllLander)"
else
echo " * $newestCommitDate: $schema"
fi

}

echo === Building schemas into $deploydir

# list of schemas to process, dependent schemas come first
schemas=(meta.yaml dialect.yaml schema.yaml schema-base.yaml)

# find the newest commit date for each schema
# publish each schema using its or any of its dependencies newest commit date.
maxDate=""
declare -A datesHash
sedCmds=()
for schema in "${schemas[@]}"; do
if [ -f "$schemaDir/$schema" ]; then
newestCommitDate=$(git log -1 --format="%ad" --date=short "$schemaDir/$schema")
@@ -23,31 +69,13 @@ for schema in "${schemas[@]}"; do
if [ "$newestCommitDate" \> "$maxDate" ]; then
maxDate=$newestCommitDate
fi
datesHash["$schema"]=$maxDate
echo $schema changed at $newestCommitDate
fi
done

# construct sed command
sedCmd=()
for schema in "${!datesHash[@]}"; do
base=$(basename "$schema" .yaml)
sedCmd+=("-e s/$base\/WORK-IN-PROGRESS/$base\/${datesHash[$schema]}/g")
done

# create the date-stamped schemas
for schema in "${!datesHash[@]}"; do
base=$(basename "$schema" .yaml)
target=deploy/oas/$version/$base/${datesHash[$schema]}

mkdir -p "deploy/oas/$version/$base"
base=$(basename $schema '.yaml')
# Add the replacement for this schema's placeholder to list of sed commands.
sedCmds+=("s/${base}\/WORK-IN-PROGRESS/${base}\/${maxDate}/g")

sed ${sedCmd[@]} $schemaDir/$schema > $target.yaml
node scripts/yaml2json/yaml2json.js $target.yaml
rm $target.yaml
mv $target.json $target

mv deploy/oas/$version/$base/*.md $target.md
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This command seems to have been lost. It is necessary when building to deploy/ for the

  • view schema/2024-11-14

links on https://spec.openapis.org to work.

Copy link
Contributor Author

@duncanbeevers duncanbeevers Feb 20, 2025 β€’

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I saw that command, but couldn't reason what it was doing, since no .md files are generated in /deploy/**.
Running the script with a clean or missing /deploy makes it a no-op.

Can you point to me where the link-updating bit happens, or show me where the .md files make their way to /deploy?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The *.md files only exist in the gh-pages branch, for example in https://github.com/OAI/OpenAPI-Specification/tree/gh-pages/oas/3.1/schema.

The title of each file is specific to that file, and I didn't want to hard-code them in the build script to minimize the differences between the OpenAPI, Overlay, and Arazzo copies of the build script.

Copy link
Contributor Author

@duncanbeevers duncanbeevers Feb 20, 2025 β€’

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'll re-add this command, but I think it indicates a weird space in the abstraction. It seems contradictory to me to both mkdir -p deploy/oas/3.1.1 and to expect that there are already markdown files present there.

I also think it's a bad pattern to do a mv where the input could be multiple files, but the destination is a single file.

It looks like the file we expect to move is actually named the same as $date. Short of restoring latest, could I just move that one $date.md file?

Alternatively, we could go back to having a latest.md, but on publish, rename it to match the latest iteration of the schema. This would preserve the publishing behavior (not creating a latest link or schema), while still removing step of manually renaming the .md file with each revision and allowing us to reference a stable filename for copying in this script.

Copy link
Contributor Author

@duncanbeevers duncanbeevers Feb 21, 2025 β€’

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've pushed a change which replicates the current *.md behavior, as well as introduces some commented-out code probing the pain points mentioned in my comment above.

  • 😐 One implementation supposes the .md file's name always matches the iteration date, which matches the current setup, and is still simpler than globbing files.

  • πŸ˜„ One implementation supposes the .md file is always latest.md. This used to be the setup, but it was abandoned. I think this approach is best. See below for more articulation.

  • 😞 One implementation mirrors the current behavior, but selects only the "first" markdown file encountered. This aligns better with the sentiment A mv command whose destination is a file should have a file as its first argument, but in a kind of nonsense way.

  • 😞 The actual implementation mirrors the current behavior, attempting to copy all the markdown files in the deploy directory to $target.md. If there are multiple markdown files present this results in an error later in the script. The actual implementation mirrors the previous one with the head -n 1 removed, to highlight its similarities to that approach.

πŸ˜„ latest.md is my favorite

The implementation which supposes a markdown file latest.md will be present is my favored approach.
🌿 indicates a benefit to the current process
🌐 indicates a benefit we already have, first gained by abandoning latest

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good discussion, please move it over to

publish_schema "$schema" "$maxDate" $(printf '%s;' "${sedCmds[@]}")
fi
done

echo === Built