Skip to content

Commit 7e2ba6b

Browse files
coder-settcoder-sett
authored
Merge pull request #17 from lishengbao/refactoring_doc
Refactoring metrics models docs
2 parents f746632 + c0d045e commit 7e2ba6b

13 files changed

+488
-38
lines changed

‎metrics-models/README.md

+20-8
Original file line numberDiff line numberDiff line change
@@ -1,31 +1,43 @@
11
# Ecosystem Evaluation System
22

3-
Ecosystem is used to describe the health status of open source community standing from ecology context. We create three top level dimensions: Productivity, Robustness and Niche Creation.
3+
Ecosystem is used to describe the health status of open source community standing from ecology context. We create a [three-dimensional space for the evaluation system](https://compass.gitee.com/zh/blog/2023/12/09/open-source-eco2/open-source-eco2), including the open source ecosystem, "collaboration, people, software" and evaluation models.
4+
5+
![image](https://github.com/oss-compass/docs/assets/53640896/49b62460-f642-4b96-89ec-7775ce81d528)
46

57
# Dimensions
68

7-
## [**Productivity**](./productivity/README.md)
9+
## [**Collaboration - Productivity**](./collaboration/productivity/README.md)
810
Definition: The efficiency with which an ecosystem or project converts inputs into output.
911

1012
Metrics Model Names | Descriptions
1113
--- | ---
12-
[Collaboration Development Index](./productivity/collaboration-development-index.md#collaboration-development-index) | Use proxy metrics to evaluate how well the development process is managed and how well the community is doing with collaborative development.
13-
[Community Service and Support](./productivity/community-service-and-support.md#community-service-and-support) | It measures the quality of services and support provided by the community as directly perceived by a developer during the contribution process.
14+
[Collaboration Development Index](./collaboration/productivity/collaboration-development-index.md#collaboration-development-index) | Use proxy metrics to evaluate how well the development process is managed and how well the community is doing with collaborative development.
15+
[Community Service and Support](./collaboration/productivity/community-service-and-support.md#community-service-and-support) | It measures the quality of services and support provided by the community as directly perceived by a developer during the contribution process.
1416

15-
## [**Robustness**](./robustness/README.md)
17+
## [**Collaboration - Robustness**](./collaboration/robustness/README.md)
1618
Definition: The capability of an ecosystem or project to face and survive disruptions.
1719

1820
Metrics Model Names | Descriptions
1921
--- | ---
20-
[Activity](./robustness/activity.md#activity) | Community Activity is used to describe how active an open source community is.
22+
[Activity](./collaboration/robustness/activity.md#activity) | Community Activity is used to describe how active an open source community is.
2123

2224

23-
## [**Niche Creation**](./niche-creation/README.md)
25+
## [**Collaboration - Niche Creation**](./collaboration/niche-creation/README.md)
2426
Definition: The capacity to create and recognize meaningful diversity and thereby new capabilities.
2527

2628
Metrics Model Names | Descriptions
2729
--- | ---
28-
[Organizations Activity](./niche-creation/ecological-diversity/organization-activity.md#organizations-activity) | It is used to describe how active organizations are in a community.
30+
[Organizations Activity](./collaboration/niche-creation/organization-activity.md#organizations-activity) | It is used to describe how active organizations are in a community.
31+
32+
## [**People-Productivity**](./people/productivity/README.md)
33+
34+
定义：The efficiency with which an ecosystem or project converts inputs into output.
35+
36+
Metrics Model Names | Descriptions
37+
--- | ---
38+
| [Contributor Domain Persona](./people/productivity/contributor-domain-persona.md) | |
39+
| [Contributor Milestone Persona](./people/productivity/contributor-milestone-persona.md) | |
40+
| [Contributor Role Persona](./people/productivity/contributor-role-persona.md) | |
2941

3042

3143
## References:

‎metrics-models/collaboration/niche-creation/README.md

+24
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,24 @@
1+
---
2+
title: Collaboration - Niche Creation
3+
slug: /dimensions-define/collaboration/niche-creation
4+
tags:
5+
- Metrics Models
6+
- Collaboration
7+
- Niche Creation
8+
description: The capacity to create and recognize meaningful diversity and thereby new capabilities.
9+
---
10+
11+
# Niche Creation
12+
13+
Definition: The capacity to create and recognize meaningful diversity and thereby new capabilities.
14+
15+
## Metrics Models
16+
17+
## [Organizations Activity](./organization-activity.md#organizations-activity)
18+
19+
Metrics Name | Definition | Threshold | Weight
20+
--- | --- | --- | ---
21+
[Org Count](./organization-activity.md#org-count) | Number of organizations to which active code contributors belong in the past 90 days. | 30 | 32.258%
22+
[Org Contributor Count](./organization-activity.md#org-contributor-count) | Number of active code contributors with organization affiliation in the past 90 days. | 300 | 25.806%
23+
[Org Commit Frequency](./organization-activity.md#org-commit-frequency) | Determine the average number of commits with organization affiliation per week in the past 90 days. | 800 | 25.806%
24+
[Org Contribution Last](./organization-activity.md#org-contribution-last) | Total contribution time of all organizations to the community in the past 90 days (weeks). | 200 | 16.129%

‎metrics-models/niche-creation/ecological-diversity/organization-activity.md renamed to ‎metrics-models/collaboration/niche-creation/organization-activity.md

+2-1
Original file line numberDiff line numberDiff line change
@@ -1,8 +1,9 @@
11
---
22
title: Organizations Activity
3-
slug: /metrics-models/niche-creation/organization-activity
3+
slug: /metrics-models/collaboration/niche-creation/organization-activity
44
tags:
55
- Metrics Models
6+
- Collaboration
67
- Niche Creation
78
- Organizations Activity
89
description: Describe how active organizations are in a community.

‎metrics-models/productivity/README.md renamed to ‎metrics-models/collaboration/productivity/README.md

+3-2
Original file line numberDiff line numberDiff line change
@@ -1,8 +1,9 @@
11
---
2-
title: Productivity
3-
slug: /dimensions-define/productivity
2+
title: Collaboration - Productivity
3+
slug: /dimensions-define/collaboration/productivity
44
tags:
55
- Metrics Models
6+
- Collaboration
67
- Productivity
78
description: The efficiency with which an ecosystem or project converts inputs into output.
89
---

‎metrics-models/productivity/collaboration-development-index.md renamed to ‎metrics-models/collaboration/productivity/collaboration-development-index.md

+2-1
Original file line numberDiff line numberDiff line change
@@ -1,8 +1,9 @@
11
---
22
title: Collaboration Development Index
3-
slug: /metrics-models/productivity/collaboration-development-index
3+
slug: /metrics-models/collaboration/productivity/collaboration-development-index
44
tags:
55
- Metrics Models
6+
- Collaboration
67
- Productivity
78
- Collaboration Development Index
89
description: Use proxy metrics to evaluate how well the development process is managed and how well the community is doing with collaborative development

‎metrics-models/productivity/community-service-and-support.md renamed to ‎metrics-models/collaboration/productivity/community-service-and-support.md

+2-1
Original file line numberDiff line numberDiff line change
@@ -1,8 +1,9 @@
11
---
22
title: Community Service and Support
3-
slug: /metrics-models/productivity/community-service-and-support
3+
slug: /metrics-models/collaboration/productivity/community-service-and-support
44
tags:
55
- Metrics Models
6+
- Collaboration
67
- Productivity
78
- Community Service and Support
89
description: Community Service and Support measures the quality of services and support provided by the community as directly perceived by a developer during the contribution process.

‎metrics-models/robustness/README.md renamed to ‎metrics-models/collaboration/robustness/README.md

+2-1
Original file line numberDiff line numberDiff line change
@@ -1,8 +1,9 @@
11
---
22
title: Robustness
3-
slug: /dimensions-define/robustness
3+
slug: /dimensions-define/collaboration/robustness
44
tags:
55
- Metrics Models
6+
- Collaboration
67
- Robustness
78
description: The capability of an ecosystem or project to face and survive disruptions.Determine
89
---

‎metrics-models/robustness/activity.md renamed to ‎metrics-models/collaboration/robustness/activity.md

+2-1
Original file line numberDiff line numberDiff line change
@@ -1,8 +1,9 @@
11
---
22
title: Activity
3-
slug: /metrics-models/robustness/activity
3+
slug: /metrics-models/collaboration/robustness/activity
44
tags:
55
- Metrics Models
6+
- Collaboration
67
- Robustness
78
- Activity
89
description: Describe how active an open source community is.

‎metrics-models/niche-creation/README.md

-23
This file was deleted.

‎metrics-models/people/productivity/README.md

+46
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,46 @@
1+
---
2+
title: People - Productivity
3+
slug: /dimensions-define/people/productivity
4+
tags:
5+
- Metrics Models
6+
- People
7+
- Productivity
8+
description: The efficiency with which an ecosystem or project converts inputs into output.
9+
---
10+
11+
# Productivity
12+
Definition: The efficiency with which an ecosystem or project converts inputs into output.
13+
14+
# Metrics Models
15+
16+
## [Contributor Domain Persona](./contributor-domain-persona.md)
17+
18+
Metrics Name | Definition | Threshold | Weight
19+
--- | --- | --- | ---
20+
| [Code Contributor Count](./contributor-domain-persona.md#Code-Contributor-Count) | How many active code class contributors in the last 90 days.| 500 | 16% |
21+
| [Code Contribution Count](./contributor-domain-persona.md#Code-Contribution-Count) | The number of contributions per capita for contributors in the active code class in the last 90 days. | 15 | 24% |
22+
| [Issue Contributor Count](./contributor-domain-persona.md#Issue-Contributor-Count) | How many active issue class contributors in the last 90 days. | 1000 | 16% |
23+
| [Issue Contribution Count](./contributor-domain-persona.md#Issue-Contribution-Count) | The number of contributions per capita for contributors in the active issue class in the last 90 days. | 10 | 24% |
24+
| [Observation Contributor Count](./contributor-domain-persona.md#Observation-Contributor-Count) | How many active observation class contributors in the last 90 days. | 2000 | 8% |
25+
| [Observation Contribution Count](./contributor-domain-persona.md#Observation-Contribution-Count) | The number of contributions per capita for contributors in the active observation class in the last 90 days. | 2 | 12% |
26+
27+
## [Contributor Milestone Persona](./contributor-milestone-persona.md)
28+
29+
Metrics Name | Definition | Threshold | Weight
30+
| --- | ---- | ------ | ------- |
31+
| [Core Contributor Count](./contributor-milestone-persona.md#Core-Contributor-Count) | How many active core contributors in the last 90 days.| 20 | 20% |
32+
| [Core Contribution Count](./contributor-milestone-persona.md#Core-Contribution-Count) | The number of contributions per capita for contributors in the active core in the last 90 days. | 200 | 30% |
33+
| [Regular Contributor Count](./contributor-milestone-persona.md#Regular-Contributor-Count) | How many active regular contributors in the last 90 days. | 100 | 12% |
34+
| [Regular Contribution Count](./contributor-milestone-persona.md#Regular-Contribution-Count) | The number of contributions per capita for contributors in the active regular in the last 90 days. | 25 | 18% |
35+
| [Visitor Contributor Count](./contributor-milestone-persona.md#Visitor-Contributor-Count) | How many active visitor contributors in the last 90 days. | 2000 | 8% |
36+
| [Visitor Contribution Count](./contributor-milestone-persona.md#Visitor-Contribution-Count) | The number of contributions per capita for contributors in the active visitor in the last 90 days. | 5 | 12% |
37+
38+
39+
## [Contributor Role Persona](./contributor-role-persona.md)
40+
41+
Metrics Name | Definition | Threshold | Weight
42+
| --- | ---- | ------ | ------- |
43+
| [Organizational Contributor Count](./contributor-role-persona.md#Organizational-Contributor-Count) | How many active organizational contributors in the last 90 days.| 1500 | 20% |
44+
| [Organizational Contribution Count](./contributor-role-persona.md#Organizational-Contribution-Count) | The number of contributions per capita for contributors in the active organizational in the last 90 days. | 10 | 30% |
45+
| [Individual Contributor Count](./contributor-role-persona.md#Individual-Contributor-Count) | How many active individual contributors in the last 90 days. | 3500 | 20% |
46+
| [Individual Contribution Count](./contributor-role-persona.md#Individual-Contribution-Count) | The number of contributions per capita for contributors in the active individual in the last 90 days. | 5 | 30% |

‎metrics-models/people/productivity/contributor-domain-persona.md

+136
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,136 @@
1+
---
2+
title: Contributor Domain Persona
3+
slug: /metrics-models/people/productivity/contributor-domain-persona
4+
tags:
5+
- Metrics Models
6+
- people
7+
- Productivity
8+
- Contributor Domain Persona
9+
description:
10+
---
11+
12+
# Contributor Domain Persona
13+
14+
In the Domain Persona Model, we first categorize the contributions made by contributors to the community by domain or type. We strive to atomize the contribution domains more finely, matching our current engineering capabilities. However, this categorization process is an ongoing evolution, and as our engineering capabilities improve, we will be able to achieve more granular domain divisions.
15+
16+
For observation-type contributions, issue-type contributions, and code-type contributions, we can use insights provided by code hosting platforms (such as GitHub, Gitee, etc.). For other types of contributions, due to differences in community usage and extent, currently we can only provide limited data sources as references. However, as our engineering capabilities gradually improve, we may provide more solutions.
17+
18+
![image](https://github.com/oss-compass/docs/assets/53640896/e83a67f0-acdc-4251-b3c9-5fbba4a0a422)
19+
20+
- Observation-type contributions: Contributors express interest and attention to a project through actions like star, fork, and watch, which can enhance and showcase the project's level of attention but do not directly contribute to the project.
21+
- Issue-type contributions: Contributions related to issues can be categorized based on the type of issues, such as usage inquiries, bug reports, task planning, etc. However, we currently lack a universal solution to identify issue types. Therefore, we classify contributions into two types based on contributor behavior in issues: regular contributions and management contributions.
22+
- Regular Issue contributions: Issue creation and comments.
23+
- Management Issue contributions: Eight types of management actions, including labeling, closing, reopening, assigning responsibility, marking as duplicate, migrating, milestones, and locking comments. While these contributions are often overlooked, they are crucial for community managers to ensure smooth issue handling, and thus, we should include them in the observation scope.
24+
- Code-type contributions: Similar to Issue-type contributions, we categorize code contributions into regular and management contributions:
25+
- Regular Code contributions: Code development, Pull Request creation, and Code Review.
26+
- Management Code contributions: Code direct submissions, along with ten types of Pull Request management actions, including labeling, closing, reopening, assigning responsibility, marking as duplicate, migrating, milestones, locking comments, merging, and reviewing, and so on. Similar to Issue management contributions, these management actions require significant effort from community managers to ensure smooth code integration.
27+
- Forum-type contributions: If an open source community builds forum services based on frameworks that provide publicly accessible data channels, such as Discourse, we can observe contributions to the forum, such as creating topics and comments.
28+
- Chat platform-type contributions: We support data inspection methods based on chat platforms like Slack, Discord, etc.
29+
- Media platform-type contributions: Currently, our data mainly comes from Twitter.
30+
- Event-type contributions: Our data currently depends on manual input, including online and offline events.
31+
- Sponsor: Cash donations.
32+
33+
34+
# Metrics in the Metrics Model
35+
36+
## Code Contributor Count
37+
38+
* Definition: How many active code class contributors in the last 90 days.
39+
* Weight: 16%
40+
* Threshold: 500
41+
42+
## Code Contribution Count
43+
44+
* Definition: The number of contributions per capita for contributors in the active code class in the last 90 days.
45+
* Weight: 24%
46+
* Threshold: 15
47+
48+
## Issue Contributor Count
49+
50+
* Definition: How many active issue class contributors in the last 90 days.
51+
* Weight: 16%
52+
* Threshold: 1000
53+
54+
## Issue Contribution Count
55+
56+
* Definition: The number of contributions per capita for contributors in the active issue class in the last 90 days.
57+
* Weight: 24%
58+
* Threshold: 10
59+
60+
## Observation Contributor Count
61+
62+
* Definition: How many active observation class contributors in the last 90 days.
63+
* Weight: 8%
64+
* Threshold: 2000
65+
66+
## Observation Contribution Count
67+
68+
* Definition: The number of contributions per capita for contributors in the active observation class in the last 90 days.
69+
* Weight: 12%
70+
* Threshold: 2
71+
72+
73+
74+
# Metric Model Algorithm
75+
76+
## Weight
77+
78+
We use [AHP](https://en.wikipedia.org/wiki/Analytic_hierarchy_process) to calculate weight of each metric.
79+
80+
### AHP Input Data
81+
82+
Metric Name | Code Contributor Count | Code Contribution Count | Issue Contributor Count | Issue Contribution Count | Observation Contributor Count | Observation Contribution Count |
83+
| --- | --- | --- | --- | --- | --- | --- |
84+
| Code Contributor Count | 1.000 | 0.667 | 1.000 | 0.667 | 2.000 | 1.333 |
85+
| Code Contribution Count | 1.500 | 1.000 | 1.500 | 1.000 | 3.000 | 2.000 |
86+
| Issue Contributor Count | 1.000 | 0.667 | 1.000 | 0.667 | 2.000 | 1.333 |
87+
| Issue Contribution Count | 1.500 | 1.000 | 1.500 | 1.000 | 3.000 | 2.000 |
88+
| Observation Contributor Count | 0.800 | 0.500 | 0.333 | 0.500 | 0.333 | 1.000 | 0.667 |
89+
| Observation Contribution Count | 0.750 | 0.500 | 0.750 | 0.500 | 1.500 | 1.000 |
90+
91+
### AHP Analysis Result
92+
93+
Metrics Name | Eigenvector | Weight
94+
--- | --- | ---
95+
| Code Contributor Count | 0.960 | 16.000% |
96+
| Code Contribution Count | 1.440 | 24.000% |
97+
| Issue Contributor Count | 0.960 | 16.000% |
98+
| Issue Contribution Count | 1.440 | 24.000% |
99+
| Observation Contributor Count | 0.480 | 8.000% |
100+
| Observation Contribution Count | 0.720 | 12.000% |
101+
102+
### Consistency Test Results
103+
104+
Largest Eigenvalue | CI Value | RI Value| CR Value | Consistency Test
105+
--- | --- | --- | --- | ---
106+
| 6.000 | 0.000 | 1.260 | 0.000 | PASS |
107+
108+
## Threshold
109+
110+
The threshold we chose is based on the big-data observations from different types of open source projects.
111+
112+
# References
113+
114+
* [Reflections on the Evaluation and Measurement of Open Source Ecosystem (1) — Evolution and Trends](https://oss-compass.org/blog/2023/12/09/open-source-eco1/open-source-eco1)
115+
* [ Reflections on the Evaluation and Measurement of Open Source Ecosystem (2) - The Multidimensional Space of Evaluation Systems](https://oss-compass.org/blog/2023/12/09/open-source-eco2/open-source-eco2)
116+
* [Reflections on the Evaluation and Measurement of Open Source Ecosystem (3) - Dynamics and Statics of Contributors](https://oss-compass.org/blog/2023/12/09/open-source-eco3/open-source-eco3)
117+
118+
# Contributors
119+
## Frontend
120+
121+
* Shengxiang Zhang
122+
* Feng Zhong
123+
* Xingyou Lai
124+
125+
## Backend
126+
127+
* Yehui Wang
128+
* Shengxiang Zhang
129+
* Shengbao Li
130+
* Huatian Qin
131+
132+
## Metric Model
133+
134+
* Yehui Wang
135+
* Liang Wang
136+
* Shengbao Li

0 commit comments

Comments
 (0)