Skip to content

refactor: consolidate TOC file handling in cloud document scripts #21511

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 8 commits into
base: master
Choose a base branch
from
+47 −72
Open

refactor: consolidate TOC file handling in cloud document scripts #21511

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
f2a10a2
refactor: consolidate TOC file handling in cloud document scripts
shhdgit Aug 5, 2025
153ee21
refactor: replace getAllMdList with getAllCloudMdList for improved TO…
shhdgit Aug 5, 2025
0ca46d9
refactor: export CLOUD_TOC_LIST for centralized TOC file management
shhdgit Aug 5, 2025
5985734
refactor: normalize TOC file links and simplify file path handling in…
shhdgit Aug 5, 2025
860bcd7
refactor: improve file path handling and link normalization in cloud …
shhdgit Aug 5, 2025
ea08a0b
refactor: update file path argument in extractFilefromList for consis…
shhdgit Aug 5, 2025
8ec9ed2
refactor: enhance file filtering logic in extractFilefromList for clo…
shhdgit Aug 5, 2025
6fbd149
refactor: streamline main function by removing console log in filterN…
shhdgit Aug 5, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 12 additions & 1 deletion scripts/filterCloudDoc.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,8 +24,19 @@ const extractFilefromList = (
});
};

const CLOUD_TOC_LIST = [
"TOC-tidb-cloud.md",
"TOC-tidb-cloud-essential.md",
"TOC-tidb-cloud-starter.md",
];
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

While it's great to consolidate the TOC file list, this CLOUD_TOC_LIST constant is now duplicated in filterCloudInitFiles.js and filterNonCloudDoc.js as well.

To avoid duplication and improve maintainability, this constant should be defined once in a shared file (e.g., scripts/utils.js) and then imported where needed. This follows the Don't Repeat Yourself (DRY) principle.

You would need to add the constant definition and export it from scripts/utils.js:

// In scripts/utils.js
export const CLOUD_TOC_LIST = [
  "TOC-tidb-cloud.md",
  "TOC-tidb-cloud-essential.md",
  "TOC-tidb-cloud-starter.md",
];

Then, you can import it here.

import { CLOUD_TOC_LIST } from "./utils.js";


const main = () => {
const filteredLinkList = getAllMdList("TOC-tidb-cloud.md");
// Get all MD lists from each TOC file and deduplicate
const allFilteredLinkLists = CLOUD_TOC_LIST.map((tocFile) =>
getAllMdList(tocFile)
);
const flattenedList = allFilteredLinkLists.flat();
const filteredLinkList = [...new Set(flattenedList)]; // Deduplicate
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

low

The comment // Deduplicate is redundant here. The [...new Set()] syntax is a common and self-documenting JavaScript idiom for creating an array with unique values.

  const filteredLinkList = [...new Set(flattenedList)];


extractFilefromList(filteredLinkList, ".", "./tmp");
copySingleFileSync("TOC-tidb-cloud.md", "./tmp/TOC.md");
Expand Down
25 changes: 15 additions & 10 deletions scripts/filterCloudInitFiles.js
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,20 +1,25 @@
import * as fs from "fs";
import path from "path";
import { getAllMdList } from "./utils.js";

// Read the TOC file
const tocContent = fs.readFileSync("TOC-tidb-cloud.md", "utf8");
const CLOUD_TOC_LIST = [
"TOC-tidb-cloud.md",
"TOC-tidb-cloud-essential.md",
"TOC-tidb-cloud-starter.md",
];
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

This CLOUD_TOC_LIST constant is duplicated across multiple scripts. To improve maintainability, it should be defined once in scripts/utils.js and imported.

Please remove this constant definition and instead modify the import on line 3 to:

import { getAllMdList, CLOUD_TOC_LIST } from "./utils.js";

You will also need to add the constant definition to scripts/utils.js and export it.


// Regular expression to match markdown links
const linkRegex = /\[([^\]]+)\]\(([^)]+)\)/g;
// Get all MD files from multiple TOCs and deduplicate
const allFilteredLinkLists = CLOUD_TOC_LIST.map((tocFile) =>
getAllMdList(tocFile)
);
const flattenedList = allFilteredLinkLists.flat();
const allFilePaths = [...new Set(flattenedList)]; // Deduplicate
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

low

The // Deduplicate comment is redundant here, as the [...new Set()] syntax is a well-known JavaScript idiom for creating an array with unique values from an iterable.

Suggested change
const allFilePaths = [...new Set(flattenedList)]; // Deduplicate
const allFilePaths = [...new Set(flattenedList)];


// Set to store unique file paths
// Set to store filtered file paths
const filePaths = new Set();

// Extract all file paths from markdown links
let match;
while ((match = linkRegex.exec(tocContent)) !== null) {
const filePath = match[2];

// Filter the file paths
for (const filePath of allFilePaths) {
// Skip external links (starting with http/https)
if (filePath.startsWith("http")) {
continue;
Expand Down
8 changes: 7 additions & 1 deletion scripts/filterNonCloudDoc.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,12 @@ import {
removeCustomContent,
} from "./utils.js";

const CLOUD_TOC_LIST = [
"TOC-tidb-cloud.md",
"TOC-tidb-cloud-essential.md",
"TOC-tidb-cloud-starter.md",
];
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

This CLOUD_TOC_LIST constant is duplicated across multiple scripts. To improve maintainability, it should be defined once in scripts/utils.js and imported.

Please remove this constant definition and instead add the following import at the top of the file with the other imports:

import { CLOUD_TOC_LIST } from "./utils.js";

You will also need to add the constant definition to scripts/utils.js and export it.


const contentHandler = (content = "") => {
return removeCustomContent("tidb", content);
};
Expand All @@ -17,7 +23,7 @@ const extractFilefromList = (
fileList.forEach((filePath = "") => {
if (
filePath.includes(`/tidb-cloud/`) ||
filePath.includes(`TOC-tidb-cloud.md`)
CLOUD_TOC_LIST.includes(filePath)
) {
return;
}
Expand Down
Loading