Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

#1448 url encoding with lower case letters #1456

Closed
wants to merge 5 commits into from
Closed

#1448 url encoding with lower case letters #1456

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
4859b59
BasicURLNormalizer.java - fix url encoding comparison
mstrewe Jan 27, 2025
7dd5bb8
BasicURLNormalizerTest.java: add test for fix
mstrewe Jan 27, 2025
e36a038
Update BasicURLNormalizerTest.java: forgot to change to example.com
mstrewe Jan 27, 2025
00c854b
BasicURLNormalizerTest.java: fix code formatting
mstrewe Jan 27, 2025
5e02c15
BasicURLNormalizer.java: simplify code and formatting
Jan 27, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion core/src/main/java/org/apache/stormcrawler/filtering/basic/BasicURLNormalizer.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -145,7 +145,7 @@ public class BasicURLNormalizer extends URLFilter {
// properly encode characters in path/file using percent-encoding
String file2 = unescapePath(file);
file2 = escapePath(file2);
if (!file.equals(file2)) {
if (!file.equalsIgnoreCase(file2)) {
hasChanged = true;
}
if (hasChanged) {
Expand Down
16 changes: 16 additions & 0 deletions core/src/test/java/org/apache/stormcrawler/filtering/BasicURLNormalizerTest.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -300,6 +300,22 @@ void testNonStandardPercentEncoding() throws MalformedURLException {
assertEquals(expectedURL, normalizedUrl, "Failed to filter query string");
}

// https://github.com/apache/incubator-stormcrawler/issues/1448
@Test
void testProperURLEncodingWithLowerCase() throws MalformedURLException {
URLFilter urlFilter = createFilter(queryParamsToFilter);
String urlWithEscapedCharacters = "http://www.example.com/Exhibitions/Detail/NjAxOA%3d%3d";
String expectedResult = "http://www.example.com/Exhibitions/Detail/NjAxOA%3d%3d";
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Shouldn't the expected result be %3D%3D?

This is the canonical representation of percent-encoded characters defined in RFC 3986.

If case variants of percent-encoded chars remain in URLs, this may cause duplicates. Note that in addition to pure lowercase variant, there could be also %3d%3D and %3D%3d.

// normalization should not change this url.
URL testSourceUrl = new URL(urlWithEscapedCharacters);
String testUrl = urlWithEscapedCharacters;
String normalizedUrl = urlFilter.filter(testSourceUrl, new Metadata(), testUrl);
assertEquals(
expectedResult,
normalizedUrl,
"Failed to normalize url encoded url with lower case letters");
}

@Test
void testHostIDNtoASCII() throws MalformedURLException {
ObjectNode filterParams = new ObjectNode(JsonNodeFactory.instance);
Expand Down
Loading