Skip to content

Commit dbd8e5d

Browse files
shadowabishadowabi
committed
update
修改Endpoint判断方法为命中规则，新增代理功能，修改说明文档
1 parent 51f47a9 commit dbd8e5d

File tree

12 files changed

+313
-179
lines changed

12 files changed

+313
-179
lines changed

‎README.md

+36-24
Original file line numberDiff line numberDiff line change
@@ -19,7 +19,7 @@ Endpoint Search is a reconnaissance tool tailored for identifying and enumeratin
1919
<a href="https://github.com/wgpsec/EndpointSearch/discussions"><strong>More Tricks »</strong></a>
2020
<br/>
2121
<br />
22-
<a href="https://github.com/wgpsec/EndpointSearch/blob/main/README.md">中文文档</a>
22+
<a href="https://github.com/wgpsec/EndpointSearch/blob/main/README_CN.md">中文文档</a>
2323
.
2424
<a href="https://github.com/wgpsec/EndpointSearch/releases">Download</a>
2525
·
@@ -31,8 +31,9 @@ Endpoint Search is a reconnaissance tool tailored for identifying and enumeratin
3131
## Features
3232
* **DNS Enumeration**: Constructs and queries DNS for probable endpoint URLs based on predefined patterns.
3333
* **SRV Record Inspection**: Automatically detects SRV records to uncover associated ports.
34-
* **HTTP/HTTPS Probing**: Tests endpoints with both HTTP and HTTPS protocols when SRV records are not present.
35-
* **Endpoint Judgment**: Determines endpoints likelihood based on response content, currently focusing on XML format.
34+
* **HTTP/HTTPS Probing**: Tests targets with both HTTP and HTTPS protocols when SRV records are not present.
35+
* **PROXY SUPPORT**: Supports HTTP/HTTPS and SOCK5 protocol proxy traffic during the HTTP request phase.
36+
* **Endpoint Judgment**: Determine the likelihood of the endpoint based on whether the response content hits the rule.
3637
* **Automatic Domain Extraction**: Extracts domains from input URLs automatically.
3738
* **Output Redundancy Removal**: Ensures unique results by deduplication.
3839
* **Configurable Behavior**: Offers a flexible configuration file for customizing service names, connection modes, and more.
@@ -44,23 +45,35 @@ Endpoint Search is a reconnaissance tool tailored for identifying and enumeratin
4445
* Alternatively, use the included Makefile to compile from source.
4546

4647
## Configuration
47-
Upon first run, the tool checks for config.json. If missing, it generates one with default settings including:
48-
* CloudService: Enumerated cloud services.
49-
* Mode & Mode2: Patterns connecting services, prefixes/suffixes, and targets.
50-
* PortList: Ports to scan if no SRV record is found.
51-
* Prefix & Suffix: Common naming conventions for prefixing or suffixing service names.
48+
When EndpointSearch is run for the first time, config.yaml and rule-yaml are detected and the default config.yaml and rule-yaml are generated if they are not present
49+
50+
config.yaml fill in as follows:
51+
```
52+
CloudService: oss,ecs
53+
Mode: .
54+
Mode2: -,.
55+
PortList: 80,443
56+
Prefix: sonic,legacy,preprod,gamma,beta,staging
57+
Suffix: sonic,legacy,preprod,gamma,beta,staging
58+
```
59+
CloudService is an enumerated cloud service name. Mode is the mode used to connect CloudService to target. Mode2 is the mode used to connect prefixes and suffixes to CloudService
60+
61+
The content of rule.yaml is as follows:
5262
```
53-
{
54-
"CloudService":"oss,ecs",
55-
"Mode":".",
56-
"Mode2" :"-,.",
57-
"PortList":"80,443",
58-
"Prefix":"sonic,legacy,preprod,gamma,beta,staging",
59-
"Suffix":"sonic,legacy,preprod,gamma,beta,staging",
60-
}
63+
rules:
64+
- Header:
65+
- text/xml
66+
- application/xml
67+
Body:
68+
- InvalidVersion
69+
- Header:
70+
- "123"
71+
Body:
72+
- ""
6173
```
74+
Multiple groups of rules can be defined. Header and Body in the Rule of each group must match exactly to be identified as endpoints. If there is only one feature, the other part can be left blank.
6275

63-
CloudService can refer to another of my dictionary projects : https://github.com/shadowabi/S-BlastingDictionary/blob/main/CloudService.txt
76+
Note that if both Header and Body in a rule are empty, all HTTP requests will pass the rule
6477

6578
## Workflow
6679
1. Enter the domain name example.com. DNS is used to enumerate example.com.
@@ -79,14 +92,12 @@ oss.example.com
7992

8093
3. If srv records already exist, HTTP/HTTPS is used to request the URL instead of enumerating the port
8194

82-
4. Otherwise, the system attempts to access the port in the target domain name + PortList through HTTP or HTTPS
95+
4. Otherwise, HTTP and HTTPS are used to access ports in the target domain name + PortList. If the proxy option is used, traffic can be forwarded to the proxy server
8396

84-
5. Finally, determine whether the entire URL is an Endpoint based on the HTTP request result. At present, determine whether the destination data is returned in xml format
97+
5. Finally, the HTTP request result determines whether the entire URL is an Endpoint, and the access is to determine whether the request traffic matches the rule in rule.yaml
8598

8699
If there are other characteristics, feel free to raise them in the Issues, or launch a PR directly.
87100

88-
The method to determine the Endpoint is implemented in the JudgeEndpoint function in the pkg directory data.go
89-
90101
## Usage
91102
```
92103
Usage:
@@ -102,17 +113,18 @@ Flags:
102113
-o, --output string 输入结果文件输出的位置 (Enter the location of the scan result output) (default "./result.txt")
103114
-p, --port string 输入需要被扫描的端口，逗号分割 (Enter the port to be scanned, separated by commas (,))
104115
--prefix string 输入枚举云服务的前缀 (Enter the prefix for enumerating the cloud service)
116+
--proxy string 使用 HTTP/SOCKS5代理，仅限web探测时 (List of http/socks5 proxy to use,Only for web detection
105117
-s, --service string 输入需要被枚举的服务名称 (Input Service Name)
106118
--suffix string 输入枚举云服务的后缀 (Enter a suffix for enumerating cloud services)
107119
-t, --timeout int 输入每个 http 请求的超时时间 (Enter the timeout period for every http request) (default 2)
108120
-u, --url string 输入目标地址 (Input [domain|url])
109121
```
110-
EndpointSearch also supports manually overwriting configuration parameters. By default, the -e parameter is CloudEndpoint in the configuration, and the -p parameter is PortList in the configuration
122+
123+
EndpointSearch can also override configuration parameters manually. For example, -e is set to CloudEndpoint by default, and -p is set to PortList by default
111124

112125
When parameters are actively specified, the default values in the configuration file are no longer used
113126

114127

115128
## TODO
116-
1. Proxy Support: Implementation of SOCKS5 proxy support for enhanced anonymity.
117-
2. Enhanced Endpoint Detection: Expanding endpoint validation criteria beyond XML responses.
129+
1. Added more ways to determine endpoints
118130

‎README_CN.md

+43-31
Original file line numberDiff line numberDiff line change
@@ -34,23 +34,35 @@
3434
或使用Makefile进行编译二进制文件后使用
3535

3636
## 配置
37-
当首次运行 EndpointSearch 时，会检测 config.json 文件是否存在，不存在则会自动创建
37+
当首次运行 EndpointSearch 时，会检测 config.yaml 和 rule.yaml 是否存在，不存在则会生成默认的 config.yaml 和 rule.yaml
3838

39-
config.json的填写内容应该如下：
39+
config.yaml 的填写内容如下：
4040
```
41-
{
42-
"CloudService":"oss,ecs",
43-
"Mode":".",
44-
"Mode2" :"-,.",
45-
"PortList":"80,443",
46-
"Prefix":"sonic,legacy,preprod,gamma,beta,staging",
47-
"Suffix":"sonic,legacy,preprod,gamma,beta,staging",
48-
}
41+
CloudService: oss,ecs
42+
Mode: .
43+
Mode2: -,.
44+
PortList: 80,443
45+
Prefix: sonic,legacy,preprod,gamma,beta,staging
46+
Suffix: sonic,legacy,preprod,gamma,beta,staging
4947
```
5048
CloudService 为枚举的云服务名称，Mode 是连接 CloudService 与 target 的方式，Mode2 是连接前后缀与 CloudService 的方式, PortList 为扫描的端口，具体例子见工作流程
5149

50+
rule.yaml 的填写内容如下：
51+
```
52+
rules:
53+
- Header:
54+
- text/xml
55+
- application/xml
56+
Body:
57+
- InvalidVersion
58+
- Header:
59+
- "123"
60+
Body:
61+
- ""
62+
```
63+
可定义多组 Rule，每组的 Rule 中的 Header 和 Body 均要完全匹配才会被判断为 Endpoint，如果仅有一种特征，可以将另一部分置空。
5264

53-
CloudService 可参考我的另一个字典项目: https://github.com/shadowabi/S-BlastingDictionary/blob/main/CloudService.txt
65+
注意，如果一个规则中的 Header 和 Body 均为空则会导致所有 HTTP 请求均会通过规则
5466

5567
## 工作流程
5668
1. 输入域名 example.com，首先会使用 DNS 去枚举 example.com，枚举方式遵循以下特点：
@@ -69,46 +81,46 @@ oss.example.com
6981

7082
3. 若已经存在 srv 记录，则不会去枚举端口，而是直接用 HTTP / HTTPS 协议去请求这个URL
7183

72-
4. 否则将通过 HTTP 和 HTTPS 协议去尝试访问目标域名 + PortList 中的端口
84+
4. 否则将通过 HTTP 和 HTTPS 协议去尝试访问目标域名 + PortList 中的端口，如果使用了代理选项，可代理流量至代理服务器
7385

74-
5. 最后通过 HTTP 的请求结果判断整个 URL 是否为 Endpoint，目前判断方式为：目标返回的数据是否为 xml 格式
86+
5. 最后通过 HTTP 的请求结果判断整个 URL 是否为 Endpoint，判断访问为判断请求流量是否命中 rule.yaml 中的规则
7587

7688
如果有其他特征，欢迎在 Issues 中提出，或者直接发起 PR。
7789

78-
判断 Endpoint 的方法在 pkg 目录 data.go 的 JudgeEndpoint 函数中实现
79-
8090
## 用法
8191
```
8292
Usage:
8393
84-
EndpointSearch [flags]
94+
EndpointSearch [flags]
8595
8696
8797
Flags:
8898
89-
-f, --file string 从文件中读取目标地址 (Input filename)
90-
-h, --help help for EndpointSearch
91-
--logLevel string 设置日志等级 (Set log level) [trace|debug|info|warn|error|fatal|panic] (default "info")
92-
-o, --output string 输入结果文件输出的位置 (Enter the location of the scan result output) (default "./result.txt")
93-
-p, --port string 输入需要被扫描的端口，逗号分割 (Enter the port to be scanned, separated by commas (,))
94-
--prefix string 输入枚举云服务的前缀 (Enter the prefix for enumerating the cloud service)
95-
-s, --service string 输入需要被枚举的服务名称 (Input Service Name)
96-
--suffix string 输入枚举云服务的后缀 (Enter a suffix for enumerating cloud services)
97-
-t, --timeout int 输入每个 http 请求的超时时间 (Enter the timeout period for every http request) (default 2)
98-
-u, --url string 输入目标地址 (Input [domain|url])
99+
-f, --file string 从文件中读取目标地址 (Input filename)
100+
-h, --help help for EndpointSearch
101+
--logLevel string 设置日志等级 (Set log level) [trace|debug|info|warn|error|fatal|panic] (default "info")
102+
-o, --output string 输入结果文件输出的位置 (Enter the location of the scan result output) (default "./result.txt")
103+
-p, --port string 输入需要被扫描的端口，逗号分割 (Enter the port to be scanned, separated by commas (,))
104+
--prefix string 输入枚举云服务的前缀 (Enter the prefix for enumerating the cloud service)
105+
--proxy string 使用 HTTP/SOCKS5代理，仅限web探测时 (List of http/socks5 proxy to use,Only for web detection
106+
-s, --service string 输入需要被枚举的服务名称 (Input Service Name)
107+
--suffix string 输入枚举云服务的后缀 (Enter a suffix for enumerating cloud services)
108+
-t, --timeout int 输入每个 http 请求的超时时间 (Enter the timeout period for every http request) (default 2)
109+
-u, --url string 输入目标地址 (Input [domain|url])
99110
```
100-
EndpointSearch 同样支持手动覆盖配置参数，-e 参数默认为配置中的 CloudEndpoint，-p 参数为配置中的 PortList
111+
112+
EndpointSearch 同样支持手动覆盖配置参数，例如 -e 参数默认为配置中的 CloudEndpoint，-p 参数为配置中的 PortList
101113

102114
当主动指定参数后，将不再使用配置文件中的默认值
103115

104116
## 功能列表
105117

106118
1. 利用 dns 服务枚举端点，隐蔽侦查
107119
2. 当域名存在时，自动探测 srv 服务发现端口
108-
3. 自动去重
109-
4. 输入的 url 将自动提取为域名
120+
3. 支持 HTTP / SOCSK5 代理
121+
4. 自动去重
122+
5. 输入的 url 将自动提取为域名
110123

111124
## TODO
112-
1. 添加 socket5 代理的支持
113-
2. 更多判断 Endpoint 的方法
125+
1. 增加更多判断 Endpoint 的方法
114126

‎cmd/rootCmd.go

+18-13
Original file line numberDiff line numberDiff line change
@@ -1,9 +1,10 @@
11
package cmd
22

33
import (
4-
"errors"
54
"fmt"
5+
"github.com/wgpsec/EndpointSearch/rule"
66
"os"
7+
"runtime"
78
"strings"
89

910
"github.com/wgpsec/EndpointSearch/define"
@@ -35,11 +36,11 @@ EndpointSearch is a scanner that probes the endpoint of a cloud service
3536
PersistentPreRun: func(cmd *cobra.Command, args []string) {
3637
log.Init(logLevel)
3738
if define.Url != "" && define.File != "" {
38-
Error.HandleFatal(errors.New("Url 参数和 File 参数不可以同时存在 (The Url parameter and File parameter cannot exist at the same time)"))
39+
Error.HandleFatal(fmt.Errorf("Url 参数和 File 参数不可以同时存在 (The Url parameter and File parameter cannot exist at the same time)"))
3940
return
4041
}
4142
if define.Url == "" && define.File == "" {
42-
Error.HandleFatal(errors.New("必选参数为空，请输入 -u 参数或 -f 参数 (The mandatory parameter is empty. Enter the -u parameter or -f parameter)"))
43+
Error.HandleFatal(fmt.Errorf("必选参数为空，请输入 -u 参数或 -f 参数 (The mandatory parameter is empty. Enter the -u parameter or -f parameter)"))
4344
return
4445
}
4546
},
@@ -72,10 +73,10 @@ EndpointSearch is a scanner that probes the endpoint of a cloud service
7273
fmt.Println("Domain srv exist:")
7374
recordList := pkg.SearchSRVRecord(ipRecordList...)
7475

75-
client := pkg.GenerateHTTPClient(define.TimeOut)
76+
client := pkg.GenerateHTTPClient(define.TimeOut, define.ProxyURL)
7677
respList := pkg.SearchEndpoint(client, portList, recordList...)
7778

78-
resultList := Compare.RemoveDuplicates(pkg.JudgeEndpoint(respList...))
79+
resultList := Compare.RemoveDuplicates(rule.JudgeEndpoint(respList...))
7980
fmt.Println("Service endpoint exist:")
8081
pkg.WriteToFile(resultList, define.OutPut)
8182
fmt.Printf("[+] The output is in %s\n", define.OutPut)
@@ -96,17 +97,21 @@ func init() {
9697
RootCmd.Flags().IntVarP(&define.TimeOut, "timeout", "t", 2, "输入每个 http 请求的超时时间 (Enter the timeout period for every http request)")
9798
RootCmd.Flags().StringVarP(&define.OutPut, "output", "o", "./result.txt", "输入结果文件输出的位置 (Enter the location of the scan result output)")
9899
RootCmd.Flags().StringVarP(&define.Port, "port", "p", "", "输入需要被扫描的端口，逗号分割 (Enter the port to be scanned, separated by commas (,))")
100+
RootCmd.Flags().StringVarP(&define.ProxyURL, "proxy", "", "", "使用 HTTP/SOCKS5代理，仅限web探测时 (List of http/socks5 proxy to use,Only for web detection")
99101
}
100102

101103
func Execute() {
102-
cc.Init(&cc.Config{
103-
RootCmd: RootCmd,
104-
Headings: cc.HiGreen + cc.Underline,
105-
Commands: cc.Cyan + cc.Bold,
106-
Example: cc.Italic,
107-
ExecName: cc.Bold,
108-
Flags: cc.Cyan + cc.Bold,
109-
})
104+
if runtime.GOOS != "windows" {
105+
cc.Init(&cc.Config{
106+
RootCmd: RootCmd,
107+
Headings: cc.HiGreen + cc.Underline,
108+
Commands: cc.Cyan + cc.Bold,
109+
Example: cc.Italic,
110+
ExecName: cc.Bold,
111+
Flags: cc.Cyan + cc.Bold,
112+
})
113+
}
114+
110115
err := RootCmd.Execute()
111116
if err != nil {
112117
os.Exit(1)

‎define/var.go

+18-8
Original file line numberDiff line numberDiff line change
@@ -9,13 +9,23 @@ type Configure struct {
99
Suffix string `mapstructure:"Suffix" json:"Suffix" yaml:"Suffix"`
1010
}
1111

12+
type Rules struct {
13+
RuleText []RuleText `mapstructure:"rules" json:"rules" yaml:"rules"`
14+
}
15+
16+
type RuleText struct {
17+
Header []string `mapstructure:"Header" json:"Header" yaml:"Header"`
18+
Body []string `mapstructure:"Body" json:"Body" yaml:"Body"`
19+
}
20+
1221
var (
13-
File string
14-
Url string
15-
Service string
16-
OutPut string
17-
TimeOut int
18-
Port string
19-
Prefix string
20-
Suffix string
22+
File string
23+
Url string
24+
Service string
25+
OutPut string
26+
TimeOut int
27+
Port string
28+
Prefix string
29+
Suffix string
30+
ProxyURL string
2131
)

‎go.mod

+1-11
Original file line numberDiff line numberDiff line change
@@ -3,38 +3,28 @@ module github.com/wgpsec/EndpointSearch
33
go 1.19
44

55
require (
6-
github.com/AlecAivazis/survey/v2 v2.3.6
7-
github.com/esonhugh/randomize v0.0.1
86
github.com/ivanpirog/coloredcobra v1.0.1
9-
github.com/olekukonko/tablewriter v0.0.5
107
github.com/sirupsen/logrus v1.9.0
118
github.com/spf13/cobra v1.5.0
129
github.com/spf13/viper v1.13.0
1310
github.com/x-cray/logrus-prefixed-formatter v0.5.2
14-
gorm.io/driver/sqlite v1.5.4
15-
gorm.io/gorm v1.25.4
11+
golang.org/x/net v0.10.0
1612
)
1713

1814
require (
1915
github.com/fatih/color v1.13.0 // indirect
2016
github.com/fsnotify/fsnotify v1.5.4 // indirect
2117
github.com/hashicorp/hcl v1.0.0 // indirect
2218
github.com/inconshreveable/mousetrap v1.0.0 // indirect
23-
github.com/jinzhu/inflection v1.0.0 // indirect
24-
github.com/jinzhu/now v1.1.5 // indirect
25-
github.com/kballard/go-shellquote v0.0.0-20180428030007-95032a82bc51 // indirect
2619
github.com/magiconair/properties v1.8.6 // indirect
2720
github.com/mattn/go-colorable v0.1.12 // indirect
2821
github.com/mattn/go-isatty v0.0.14 // indirect
29-
github.com/mattn/go-runewidth v0.0.13 // indirect
30-
github.com/mattn/go-sqlite3 v1.14.17 // indirect
3122
github.com/mgutz/ansi v0.0.0-20200706080929-d51e80ef957d // indirect
3223
github.com/mitchellh/mapstructure v1.5.0 // indirect
3324
github.com/onsi/ginkgo v1.16.5 // indirect
3425
github.com/onsi/gomega v1.20.2 // indirect
3526
github.com/pelletier/go-toml v1.9.5 // indirect
3627
github.com/pelletier/go-toml/v2 v2.0.5 // indirect
37-
github.com/rivo/uniseg v0.2.0 // indirect
3828
github.com/rogpeppe/go-internal v1.8.0 // indirect
3929
github.com/spf13/afero v1.8.2 // indirect
4030
github.com/spf13/cast v1.5.0 // indirect

0 commit comments

Comments
 (0)