Skip to content

Commit

Permalink
fix(release): Refactor release script (#5234)
Browse files Browse the repository at this point in the history 
* fix python errors in release schript

* skip site validation unless Applitools API key is present

* modify search config to include kinetics section and component blueprint pages

* add a README for search

* update index_name in search config

* more searchconfig edits

* update index name

* revert back to use localhost

* adjust existing dist prompt

* fix invalid json in searchconfig

* refactor search config and release script to use docker

* add explicit check for make-index flag to opt-in to making site search index

* display error message if .algoliakeys.json is missing

* Update README-search.md
  • Loading branch information
@Dottenpixel
Dottenpixel authored Feb 1, 2024
1 parent 99d5420 commit 4b8f090
Show file tree
Hide file tree
Showing 3 changed files with 121 additions and 71 deletions.
29 changes: 29 additions & 0 deletions README-search.md
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,29 @@
# Salesforce Lightning Design System

Welcome to the [Salesforce Lightning Design System](https://www.lightningdesignsystem.com) brought to you by [Salesforce UX](https://twitter.com/salesforceux).

* Tailored for building Salesforce apps: Using the Lightning Design System markup and CSS framework results in UIs that reflect the Salesforce Lightning look and feel.
* Continuously updated: As long as you’re using the latest version of the Lightning Design System, your pages are always up to date with Salesforce UI changes.

## Search

Handled by Algolia.
This process decribes indexing as a standalone process. Search indexing can also be done at time of site build by passing `--make-index` to the script: `sh ./release.sh --make-index`

To crawl and update the index, use the [Docker method](https://docsearch.algolia.com/docs/legacy/run-your-own/#run-the-crawl-from-the-docker-image).

First, create the `.env` file.:
```
echo APPLICATION_ID=XXXXXX >> .env
echo API_KEY=0123456789 >> .env
```
Access the Algolia account, or ask someone who has access, to get the correct values.

Then install Docker Desktop as described [here](https://stackoverflow.com/a/44719239/390866):
`brew install --cask docker`

Then launch the Docker app. Click next. It will ask for privileged access. Confirm. A whale icon should appear in the top bar. Click it, and wait for "Docker is running" to appear.

Then in terminal in the `design-system-internal` directory:

`docker run -it --env-file=.env -e "CONFIG=$(cat ./searchconfig.json | jq -r tostring)" algolia/docsearch-scraper`
118 changes: 61 additions & 57 deletions release.sh
View file
Original file line number Diff line number Diff line change
Expand Up @@ -9,25 +9,45 @@ package_json="$dist_path/package.json"

use_exising_dist=false

echo "==============================================================="
if [[ "$@" == "--make-index" ]]; then
make_index=true
echo "» Site index to be built"

# Check if the .algoliakeys.json file exists
if [ ! -f ".algoliakeys.json" ]; then
# Print an error message
tput setab 1; echo "» Error: .algoliakeys.json file not found and is needed to index the site"; tput sgr0
# Exit the script with a non-zero status code
exit 1
fi
else
make_index=false
echo "» Skipping site index build"
fi
echo "==============================================================="


if [ -e "$package_json" ]; then
# Use jq to extract the version property
version=$(jq -r '.version' "$package_json")

# Prompt the user to continue
read -p "The version v$version exists in the .dist folder. Do you want to continue? (y/n): " choice
read -p "The version v$version exists in the .dist folder. Do you want to continue? (Y/n)" choice
choice=${choice:-y} # set default value to "y" if user presses Enter

# Check the user's choice
if [[ "$choice" == "y" || "$choice" == "Y" ]]; then
echo "==============================================================="
echo "Continuing with the existing version $version"
echo "» Continuing with the existing version $version"
echo "==============================================================="
use_exising_dist=true
fi
fi

if [ "$use_exising_dist" = false ]; then
echo "==============================================================="
echo "Continuing to build and dist anew from v$(jq -r '.version' "./package.json")"
echo "» Continuing to build and dist anew from v$(jq -r '.version' "./package.json")"
echo "==============================================================="
fi

Expand Down Expand Up @@ -137,73 +157,57 @@ if [ "$build_site_only" = false ]; then
# Publish the tarball to the Heroku app
heroku builds:create --source-tar site-release.tar.gz -a ${HEROKU_APP_NAME}

# Clone Algolia Docsearch Scraper
echo "» Installing docsearch scraper..."
git clone https://github.com/algolia/docsearch-scraper.git

cd ./.www/
if [ "$make_index" = true ]; then
echo "» Running docsearch scraper from Docker..."

# start python server on port 80
python3 -m http.server 80 &

# install python3
brew install python

cd ../docsearch-scraper/
# start python server on port 80
python3 -m http.server 80 &

# crudely install pipenv
curl https://raw.githubusercontent.com/pypa/pipenv/master/get-pipenv.py | python
# copy Algolia environment variables into .env
# relies on design-system-site/environment.js being run first
echo APPLICATION_ID=$(cat ../../.algoliakeys.json | jq -r .SLDS__SEARCH__APP_ID) > .env
echo API_KEY=$(cat ../../.algoliakeys.json | jq -r .SLDS__SEARCH__API_KEY) >> .env

# copy Algolia environment variables into .env
# relies on design-system-site/environment.js being run first
echo APPLICATION_ID=$(cat ../.algoliakeys.json | jq -r .SLDS__SEARCH__APP_ID) > .env
echo API_KEY=$(cat ../.algoliakeys.json | jq -r .SLDS__SEARCH__API_KEY) >> .env
# Run Algolia Docsearch Scraper in Docker
# https://docsearch.algolia.com/docs/legacy/run-your-own/#run-the-crawl-from-the-docker-image
docker run -v /dev/shm:/dev/shm -it --env-file=.env -e "CONFIG=$(cat ../../searchconfig.json | jq -r tostring)" algolia/docsearch-scraper


# install pipenv dependencies
pipenv install

# check for active python server on port 80
max_iterations=10
wait_seconds=1
http_endpoint="http://127.0.0.1:80/"
iterations=0

while true
do
((iterations++))
echo "» Attempt $iterations"
sleep $wait_seconds
http_code=$(curl --verbose -s -o /tmp/result.txt -w '%{http_code}' "$http_endpoint";)

# python server is running, begin crawling
if [ "$http_code" -eq 200 ]; then
echo "» Python server active. Crawling site..."
pipenv run ./docsearch run ../../searchconfig.json
break
fi

# no active server, end loop
if [ "$iterations" -ge "$max_iterations" ]; then
echo "» No active python server. Skipping indexing operation..."
exit 1
fi
done

# kill python server
lsof -ti tcp:80 | xargs kill
# kill python server
lsof -ti tcp:80 | xargs kill
fi

# Exit back to parent directory and clean-up after ourselves
cd ../../

echo "» Removing '__release' folder..."
rm -rf __release/

# Prompt the user to remove release folder
read -p "Do you want to delete the '__release' folder? (y/n): " do_delete_release_folder

# Check the user's choice
if [[ "$do_delete_release_folder" == "y" || "$do_delete_release_folder" == "Y" ]]; then
echo "==============================================================="
echo "» Removing __release folder..."
echo "==============================================================="
rm -rf __release/
fi

cp postcss.config.js.bak postcss.config.js
rm postcss.config.js.bak

# Validate staged site
echo "» Validating site..."
SLDS_VALIDATION_URL=${VALIDATION_URL} npx ava __tests__/site/site-validation.ava.js
# Prompt the user to remove release folder
read -p "Do you want to validate the site? (y/n): " do_validate_site

# Check the user's choice
if [[ "$do_validate_site" == "y" || "$do_validate_site" == "Y" ]]; then
echo "==============================================================="
echo "» Validating site..."
echo "==============================================================="
# Validate staged site
SLDS_VALIDATION_URL=${VALIDATION_URL} npx ava __tests__/site/site-validation.ava.js
fi

fi

# Exit back to parent directory
Expand Down
45 changes: 31 additions & 14 deletions searchconfig.json
View file
Original file line number Diff line number Diff line change
Expand Up @@ -2,59 +2,76 @@
"index_name": "summer-24",
"start_urls": [
{
"url": "http://localhost/components/overview",
"url": "http://host.docker.internal/components/overview",
"page_rank": 2
},
{
"url": "http://localhost/design-tokens/",
"url": "http://host.docker.internal/design-tokens/",
"page_rank": 1
},
{
"url": "http://localhost/icons/",
"url": "http://host.docker.internal/icons/",
"page_rank": 1
},
{
"url": "http://localhost/guidelines/",
"url": "http://host.docker.internal/guidelines/",
"page_rank": -1
},
{
"url": "http://localhost/faq/",
"url": "http://host.docker.internal/kinetics/",
"page_rank": -1
},
{
"url": "http://host.docker.internal/resources/faq/",
"page_rank": -2
},
{
"url": "http://localhost/platforms/",
"url": "http://host.docker.internal/platforms/",
"page_rank": -2
},
{
"url": "http://localhost/getting-started/",
"url": "http://host.docker.internal/getting-started/",
"page_rank": -2
},
{
"url": "http://localhost/release-notes/",
"url": "http://host.docker.internal/release-notes/",
"page_rank": -3
},
{
"url": "http://localhost/articles/",
"url": "http://host.docker.internal/resources/articles/",
"page_rank": -3
},
{
"url": "http://localhost/downloads/",
"url": "http://host.docker.internal/resources/downloads/",
"page_rank": -3
},
{
"url": "http://localhost/",
"url": "http://host.docker.internal/",
"page_rank": 1
}
],
"separatorsToIndex": ".-_",
"stop_urls": [],
"stop_urls": [
"http://host.docker.internal/components/.*?/\\?variant.*?",
"http://host.docker.internal/components/.*?/\\?modifiers.*?",
"http://host.docker.internal/components/.*?/\\?example.*?"
],
"selectors_exclude": ["span.slds-badge", ".docsearch-ignore"],
"selectors": {
"lvl0": ".docsearch-category",
"lvl1": "h1",
"lvl2": ".docsearch-level-2",
"text": ".site-main-stage p, .docsearch-text, .slds-text-longform p, .site-content th"
"lvl2": {
"selector": ".docsearch-level-2, .site-main-stage h2",
"strip_chars": "#"
},
"lvl3": {
"selector": ".docsearch-level-3, .site-main-stage h3",
"strip_chars": "#"
},
"text": ".site-main-stage p, .site-main-stage__content li, .site-main-stage figure, .docsearch-text, .slds-text-longform p, .site-content th"
},
"min_indexed_level": 1,
"js_render": true,
"js_wait": 1,
"nb_hits": 19300
}

0 comments on commit 4b8f090

Please sign in to comment.