Skip to content

Replace existing codebase with new-log-viewer and update docs. #101

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 23 commits into from
Oct 24, 2024
Merged

Replace existing codebase with new-log-viewer and update docs. #101

Show file tree
Hide file tree
Changes from 13 commits
Commits
Show all changes
23 commits
Select commit Hold shift + click to select a range
b867f00
Remove existing code to prepare for new log viewer.
junhaoliao Oct 22, 2024
5bdce76
Move new log viewer code to project root.
junhaoliao Oct 22, 2024
baf68a7
Update paths in linting GitHub workflow.
junhaoliao Oct 22, 2024
5cef2df
Update docs.
junhaoliao Oct 22, 2024
b758fdc
Remove pretty print description.
junhaoliao Oct 22, 2024
863c1a2
Add prettify into "Features in Development".
junhaoliao Oct 22, 2024
0ba566a
Docs - Apply suggestions from code review
junhaoliao Oct 22, 2024
7f87e26
Apply `# How does it work?` docs suggestions from Marco.
junhaoliao Oct 22, 2024
112f89a
Move "How does it work?" section to a more logical position in the RE…
junhaoliao Oct 22, 2024
8f2e55d
Refactor README.md to use reference-style links.
junhaoliao Oct 23, 2024
06258f5
Update monaco editor link.
junhaoliao Oct 23, 2024
04291bc
Move dev docs into docs/dev-guide.
junhaoliao Oct 23, 2024
f15a846
Reformat docs.
junhaoliao Oct 23, 2024
a704722
Edit README.
kirkrodrigues Oct 23, 2024
54d8d1c
Edit README again.
kirkrodrigues Oct 23, 2024
a8c0e6e
Docs - Apply suggestions by Kirk from code review
junhaoliao Oct 23, 2024
afeaec0
Docs - Apply suggestions by Marco from code review
junhaoliao Oct 23, 2024
172b346
Remove "no larger than 2GB uncompressed" descritpion from "Features i…
junhaoliao Oct 23, 2024
a6087c7
Reflow docs to achieve max line length of 100.
junhaoliao Oct 23, 2024
3183de0
In README add instructions for starting a local server.
junhaoliao Oct 24, 2024
9288d84
Fix list indent in validation.md.
junhaoliao Oct 24, 2024
9f0bb69
Turn md references into clickable links - Apply suggestions from code…
junhaoliao Oct 24, 2024
5ec9772
Add missing grammatical article - Apply suggestions from code review
junhaoliao Oct 24, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 0 additions & 3 deletions .babelrc
View file
Open in desktop

This file was deleted.

70 changes: 0 additions & 70 deletions .eslintrc.js
View file
Open in desktop

This file was deleted.

4 changes: 2 additions & 2 deletions .github/workflows/lint.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -28,5 +28,5 @@ jobs:
- uses: "actions/setup-node@v4"
with:
node-version: 22
Comment on lines 28 to 30
Copy link

@coderabbitai coderabbitai bot Oct 22, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue

Consider using LTS version of Node.js

Node.js 22 is currently in pre-release and not recommended for production use. Consider using Node.js 20 (LTS) for better stability.

Apply this diff:

       - uses: "actions/setup-node@v4"
         with:
-          node-version: 22
+          node-version: 20
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
- uses: "actions/setup-node@v4"
with:
node-version: 22
- uses: "actions/setup-node@v4"
with:
node-version: 20

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Node.js 22 has been released as LTS on Apr 24, 2024

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@junhaoliao, you're correct. Node.js 22 has been released as LTS as of April 2024. Thank you for the clarification.

(_/)
(•_•)

✏️ Learnings added 
Learnt from: junhaoliao
PR: y-scope/yscope-log-viewer#101
File: .github/workflows/lint.yaml:28-30
Timestamp: 2024-10-24T01:21:44.106Z
Learning: As of April 2024, Node.js 22 is LTS.

Note: Learnings are effective only in the context of similar code segments. To apply general coding guidelines, please configure review instructions. You can manage existing learnings in the app.

- run: "npm --prefix new-log-viewer/ clean-install"
- run: "npm --prefix new-log-viewer/ run lint:ci"
- run: "npm clean-install"
- run: "npm run lint:ci"
10 changes: 5 additions & 5 deletions .gitignore
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,12 +1,12 @@
# dependencies
# Dependencies
/node_modules

# production
# Build
/dist

# For bundle size analysis
/bundle-stats.json

# IDEs
/.idea
/.vscode

# for bundle size analysis
/bundle_stats.json
0 new-log-viewer/.prettierrc.yaml → .prettierrc.yaml
View file
Open in desktop
File renamed without changes.
0 new-log-viewer/.stylelintrc.json → .stylelintrc.json
View file
Open in desktop
File renamed without changes.
147 changes: 54 additions & 93 deletions README.md
View file
Open in desktop
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Remove petty print feature from intro.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

right, not sure how i was thinking about syntax highlighting when i first looked at that lol

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I guess we could leave the online demo there..., but this comment is just a reminder that we should replace the demo at some point. Maybe we add an issue?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

#102

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fix link to Golang library. It does actually exist now.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was thinking clp-ffi-go is not a logging library so i intentionally left it. Do we want to add clp-ffi-go?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh ur correct. Then maybe we just delete it. I don't know when this is coming...

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Discussed offline that we might want to keep the line

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The link here "More info on the dataset and other datasets can be found here." is broken

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I still thinks validation should be in dev docs. If you disagree let me know. I guess we can leave the how it works in main README

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I get what you mean now. Let's move some of those into dev-guide.

Original file line number Diff line number Diff line change
@@ -1,124 +1,85 @@
# YScope Log Viewer

[![Open bug reports](https://img.shields.io/github/issues/y-scope/yscope-log-viewer/bug?label=bugs)](https://github.com/y-scope/yscope-log-viewer/issues?q=is%3Aissue+is%3Aopen+label%3Abug)
[![Open feature requests](https://img.shields.io/github/issues/y-scope/yscope-log-viewer/enhancement?label=feature-requests)](https://github.com/y-scope/yscope-log-viewer/issues?q=is%3Aissue+is%3Aopen+label%3Aenhancement)
[![CLP on Zulip](https://img.shields.io/badge/zulip-yscope--clp%20chat-1888FA?logo=zulip)](https://yscope-clp.zulipchat.com/)
![Open bug reports][bugs-shield]
![Open feature requests][feature-requests-shield]
![CLP on Zulip][zulip-shield]

`yscope-log-viewer` is a tool that can be used to view log files that were
compressed using [CLP](https://github.com/y-scope/clp)'s IR stream format. The
viewer can be used to navigate the log file, filter by log level, pretty print
the logs and generate direct links to specific log events.
`yscope-log-viewer` is a tool that can be used to view log files that were
compressed using [CLP][clp-repo]'s IR stream format. It also supports viewing
JSON logs. The viewer can be used to navigate the log file, filter by log level,
view the logs with syntax highlights and generate direct links to specific log
events.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
`yscope-log-viewer` is a tool that can be used to view log files that were
compressed using [CLP][clp-repo]'s IR stream format. It also supports viewing
JSON logs. The viewer can be used to navigate the log file, filter by log level,
view the logs with syntax highlights and generate direct links to specific log
events.
Yscope Log Viewer is a browser tool for viewing compressed [CLP][clp-repo] IR log files and JSON logs. It allows users to easily navigate through log files, filter by log level, and create direct links to specific log events.


See the [features in development](#features-in-development) section for upcoming
features.

# Online Demo

* A demo of the log viewer can be found at https://yscope.com/log-viewer
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue

Fix bare URL in online demo section.

The URL should be properly formatted as a Markdown link for consistency and to follow Markdown best practices.

Apply this change:

-* A demo of the log viewer can be found at https://yscope.com/log-viewer
+* A demo of the log viewer can be found at [https://yscope.com/log-viewer](https://yscope.com/log-viewer)
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
* A demo of the log viewer can be found at https://yscope.com/log-viewer
* A demo of the log viewer can be found at [https://yscope.com/log-viewer](https://yscope.com/log-viewer)
🧰 Tools
🪛 Markdownlint

17-17: null
Bare URL used

(MD034, no-bare-urls)

* The demo loads a Hadoop YARN log file from the
[hive-24hrs](https://zenodo.org/record/7094921#.Y5JbH33MKHs) log dataset.
* More info on the dataset and other datasets can be found
[here](https://github.com/y-scope/clp/blob/main/docs/Datasets.md).
* To open an IR stream, drag and drop it onto the log viewer or use the open
* The demo loads a Hadoop YARN log file from the [hive-24hrs][hive-24hrs] log
dataset.
* More info on the dataset and other datasets can be found [here][datasets].
* To open an IR stream, drag and drop it onto the log viewer or use the open
file dialog.

# Generating IR Stream Logs

IR stream log files can currently be generated using these libraries:

* [Python Logging Library](https://github.com/y-scope/clp-loglib-py)
* [Log4j Logging Library](https://github.com/y-scope/log4j1-appenders)
* [Log4j Logging Library][log4j1-appenders]
* [Logback Logging Library][logback-appenders]
* Golang Logging Library (in development)

# Providing Feedback

You can use GitHub issues to [report a bug](https://github.com/y-scope/yscope-log-viewer/issues/new?assignees=&labels=bug&template=bug-report.yml)
or [request a feature](https://github.com/y-scope/yscope-log-viewer/issues/new?assignees=&labels=enhancement&template=feature-request.yml).

Join us on [Zulip](https://yscope-clp.zulipchat.com/) to chat with developers
and other community members.

# Developing `yscope-log-viewer`

* Clone the repo to get a copy of the code for development

```shell
git clone https://github.com/yscope/yscope-log-viewer.git
cd yscope-log-viewer
```

* Install Node.js using a [release](https://nodejs.org/en/) or via
[nvm](https://github.com/nvm-sh/nvm)
* Install the dependencies:

```shell
$ npm install
```

* Run the development server:

```shell
$ npm start
```

* The application should now be served at http://localhost:3010.

# Distribute

To create a build, run the following command and the build will be placed in the
`dist` folder:

```shell
$ npm run build
```
* [Python Logging Library][clp-loglib-py]

# How does it work?

`yscope-log-viewer` is written using the ReactJS framework and uses the open
source [monaco-editor](https://github.com/microsoft/monaco-editor).
`yscope-log-viewer` is written using the ReactJS framework and uses the open
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
`yscope-log-viewer` is written using the ReactJS framework and uses the open
Yscope Log Viewer is written using the ReactJS framework and uses the open

source [monaco-editor][monaco-editor]. [clp-ffi-js][clp-ffi-js] is used to
decode CLP IR files.

To open IR stream files, the viewer spawns workers to do the following:
The log viewer spawns a worker to assist with computationally intensive tasks
such as:

* Decompress the Zstd-compressed file
* Build an index of log events
* Paginate the indexed logs based on the number of log events per page
* Decode the CLP-encoded log data as needed
* Deserializing the file and creating an index of logs events.
* Paginating the indexed logs.
* Decoding the deserialized events log events into plain-text.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
* Decoding the deserialized events log events into plain-text.
* Decoding the deserialized events log events into plain text.


Once the worker decompresses, decodes, and extracts logs, the viewer UI can be
used to navigate the log. Tasks are passed to the worker as needed and changes
are rendered to the UI.
Tasks are passed to the worker as needed and changes are rendered in the UI.

# Validation

Currently, there is limited automated testing to verify that changes don't cause
bugs. While this is being developed, the following tests can be run manually:
# Providing Feedback

* Verify that the following features work:
* Changing the number of events per page
* Navigating to the first/last/next/previous page
* Loading a log file using the open file dialog and dragging & dropping
* Copying a link to a log event
* Change the log level
* Prettifying logs
* Using the keyboard shortcuts
* Perform a build and verify that all features are functional
You can use GitHub issues to [report a bug][report-bug] or
[request a feature][report-enhancement].

## Running tests
Join us on [Zulip][zulip] to chat with developers and other community members.

We use [`jest`](https://jestjs.io/docs/getting-started) as our testing
framework. Tests can be run with:
# Developer Guide

```shell
npm test
```
You can find our docs source in `docs/dev-guide`.

# Features in Development

* Dashboard to visualize distribution of log types and log levels
* Opening log files which are larger than 2 GB when uncompressed
* Support for plain text, archived log files, and other requested formats
* Search across file, multiple files or within provided time ranges
* Infinite scrolling instead of using pagination
* Log correlation with sync by timestamp across multiple editors
* Automatic conversion of text log files to IR stream format in the browser
* Deployment of components via NPM
* Pretty print to enhance readability of structured data in the logs.
* Dashboard to visualize distribution of log types and log levels.
* Opening log files which are larger than 2 GB when uncompressed.
* Support for plain text, archived log files, and other requested formats.
* Search across file, multiple files or within provided time ranges.
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue

Fix grammar in features list.

Add Oxford comma for clarity in the list of search capabilities.

Apply this change:

-* Search across file, multiple files or within provided time ranges.
+* Search across file, multiple files, or within provided time ranges.
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
* Search across file, multiple files or within provided time ranges.
* Search across file, multiple files, or within provided time ranges.
🧰 Tools
🪛 LanguageTool

[uncategorized] ~63-~63: Possible missing comma found.
Context: ...formats. * Search across file, multiple files or within provided time ranges. * Infin...

(AI_HYDRA_LEO_MISSING_COMMA)

* Infinite scrolling instead of using pagination.
* Log correlation with sync by timestamp across multiple editors.
* Automatic conversion of text log files to IR stream format in the browser.
* Deployment of components via NPM.

[bugs-shield]: https://img.shields.io/github/issues/y-scope/yscope-log-viewer/bug?label=bugs
[clp-ffi-js]: https://github.com/y-scope/clp-ffi-js
[clp-loglib-py]: https://github.com/y-scope/clp-loglib-py
[clp-repo]: https://github.com/y-scope/clp
[datasets]: https://docs.yscope.com/clp/main/user-guide/resources-datasets
[feature-requests-shield]: https://img.shields.io/github/issues/y-scope/yscope-log-viewer/enhancement?label=feature-requests
[hive-24hrs]: https://zenodo.org/record/7094921#.Y5JbH33MKHs
[log4j1-appenders]: https://github.com/y-scope/log4j1-appenders
[logback-appenders]: https://github.com/y-scope/logback-appenders
[monaco-editor]: https://microsoft.github.io/monaco-editor/
[report-bug]: https://github.com/y-scope/yscope-log-viewer/issues/new?labels=bug&template=bug-report.yml
[report-enhancement]: https://github.com/y-scope/yscope-log-viewer/issues/new?labels=enhancement&template=feature-request.yml
[zulip]: https://yscope-clp.zulipchat.com/
[zulip-shield]: https://img.shields.io/badge/zulip-yscope--clp%20chat-1888FA?logo=zulip
8 changes: 0 additions & 8 deletions customized-packages/zstd-codec/CHANGELOG.md
View file
Open in desktop

This file was deleted.

42 changes: 0 additions & 42 deletions customized-packages/zstd-codec/Dockerfile
View file
Open in desktop

This file was deleted.

8 changes: 0 additions & 8 deletions customized-packages/zstd-codec/LICENSE
View file
Open in desktop

This file was deleted.

Loading
Loading