diff --git a/doc/benchmarks/flavors/article_light/benchmaking-biorxiv.md b/doc/benchmarks/flavors/article_light/benchmaking-biorxiv.md index 7ed8e68f68..9c5216c93f 100644 --- a/doc/benchmarks/flavors/article_light/benchmaking-biorxiv.md +++ b/doc/benchmarks/flavors/article_light/benchmaking-biorxiv.md @@ -7,14 +7,14 @@ Evaluation on 1996 random PDF files out of 1998 PDF (ratio 1.0). **Field-level results** -| label | precision | recall | f1 | support | -|--- |--- |--- |--- |--- | -| authors | 82.99 | 81.45 | 82.22 | 1995 | -| first_author | 96.32 | 94.63 | 95.47 | 1993 | -| title | 78.19 | 73.65 | 75.85 | 1996 | -| | | | | | -| **all fields (micro avg.)** | **85.94** | **83.24** | **84.57** | 5984 | -| all fields (macro avg.) | 85.84 | 83.24 | 84.51 | 5984 | +| label | precision | recall | f1 | support | +|-----------------------------|-----------|-----------|-----------|---------| +| authors | 82.99 | 81.45 | 82.22 | 1995 | +| first_author | 96.32 | 94.63 | 95.47 | 1993 | +| title | 78.19 | 73.65 | 75.85 | 1996 | +| | | | | | +| **all fields (micro avg.)** | **85.94** | **83.24** | **84.57** | 5984 | +| all fields (macro avg.) | 85.84 | 83.24 | 84.51 | 5984 | @@ -22,14 +22,14 @@ Evaluation on 1996 random PDF files out of 1998 PDF (ratio 1.0). **Field-level results** -| label | precision | recall | f1 | support | -|--- |--- |--- |--- |--- | -| authors | 83.55 | 82.01 | 82.77 | 1995 | -| first_author | 96.63 | 94.93 | 95.77 | 1993 | -| title | 80.64 | 75.95 | 78.22 | 1996 | -| | | | | | -| **all fields (micro avg.)** | **87.03** | **84.29** | **85.64** | 5984 | -| all fields (macro avg.) | 86.94 | 84.3 | 85.59 | 5984 | +| label | precision | recall | f1 | support | +|-----------------------------|-----------|-----------|-----------|---------| +| authors | 83.55 | 82.01 | 82.77 | 1995 | +| first_author | 96.63 | 94.93 | 95.77 | 1993 | +| title | 80.64 | 75.95 | 78.22 | 1996 | +| | | | | | +| **all fields (micro avg.)** | **87.03** | **84.29** | **85.64** | 5984 | +| all fields (macro avg.) | 86.94 | 84.3 | 85.59 | 5984 | @@ -37,14 +37,14 @@ Evaluation on 1996 random PDF files out of 1998 PDF (ratio 1.0). **Field-level results** -| label | precision | recall | f1 | support | -|--- |--- |--- |--- |--- | -| authors | 91.57 | 89.87 | 90.72 | 1995 | -| first_author | 96.78 | 95.08 | 95.93 | 1993 | -| title | 92.13 | 86.77 | 89.37 | 1996 | -| | | | | | -| **all fields (micro avg.)** | **93.51** | **90.57** | **92.02** | 5984 | -| all fields (macro avg.) | 93.49 | 90.58 | 92 | 5984 | +| label | precision | recall | f1 | support | +|-----------------------------|-----------|-----------|-----------|---------| +| authors | 91.57 | 89.87 | 90.72 | 1995 | +| first_author | 96.78 | 95.08 | 95.93 | 1993 | +| title | 92.13 | 86.77 | 89.37 | 1996 | +| | | | | | +| **all fields (micro avg.)** | **93.51** | **90.57** | **92.02** | 5984 | +| all fields (macro avg.) | 93.49 | 90.58 | 92 | 5984 | @@ -52,14 +52,14 @@ Evaluation on 1996 random PDF files out of 1998 PDF (ratio 1.0). **Field-level results** -| label | precision | recall | f1 | support | -|--- |--- |--- |--- |--- | -| authors | 87.59 | 85.96 | 86.77 | 1995 | -| first_author | 96.32 | 94.63 | 95.47 | 1993 | -| title | 88.35 | 83.22 | 85.71 | 1996 | -| | | | | | -| **all fields (micro avg.)** | **90.79** | **87.93** | **89.34** | 5984 | -| all fields (macro avg.) | 90.75 | 87.94 | 89.32 | 5984 | +| label | precision | recall | f1 | support | +|-----------------------------|-----------|-----------|-----------|---------| +| authors | 87.59 | 85.96 | 86.77 | 1995 | +| first_author | 96.32 | 94.63 | 95.47 | 1993 | +| title | 88.35 | 83.22 | 85.71 | 1996 | +| | | | | | +| **all fields (micro avg.)** | **90.79** | **87.93** | **89.34** | 5984 | +| all fields (macro avg.) | 90.75 | 87.94 | 89.32 | 5984 | #### Instance-level results diff --git a/doc/benchmarks/flavors/article_light/benchmaking-elife.md b/doc/benchmarks/flavors/article_light/benchmaking-elife.md index 4e3256c650..d6c7637c4b 100644 --- a/doc/benchmarks/flavors/article_light/benchmaking-elife.md +++ b/doc/benchmarks/flavors/article_light/benchmaking-elife.md @@ -7,14 +7,14 @@ Evaluation on 957 random PDF files out of 982 PDF (ratio 1.0). **Field-level results** -| label | precision | recall | f1 | support | -|--- |--- |--- |--- |--- | -| authors | 79.68 | 77.43 | 78.54 | 957 | -| first_author | 91.83 | 89.33 | 90.56 | 956 | -| title | 89.25 | 85.89 | 87.54 | 957 | -| | | | | | -| **all fields (micro avg.)** | **86.91** | **84.22** | **85.54** | 2870 | -| all fields (macro avg.) | 86.92 | 84.22 | 85.55 | 2870 | +| label | precision | recall | f1 | support | +|-----------------------------|-----------|-----------|-----------|---------| +| authors | 79.68 | 77.43 | 78.54 | 957 | +| first_author | 91.83 | 89.33 | 90.56 | 956 | +| title | 89.25 | 85.89 | 87.54 | 957 | +| | | | | | +| **all fields (micro avg.)** | **86.91** | **84.22** | **85.54** | 2870 | +| all fields (macro avg.) | 86.92 | 84.22 | 85.55 | 2870 | @@ -22,14 +22,14 @@ Evaluation on 957 random PDF files out of 982 PDF (ratio 1.0). **Field-level results** -| label | precision | recall | f1 | support | -|--- |--- |--- |--- |--- | -| authors | 80 | 77.74 | 78.86 | 957 | -| first_author | 91.83 | 89.33 | 90.56 | 956 | -| title | 96.42 | 92.79 | 94.57 | 957 | -| | | | | | -| **all fields (micro avg.)** | **89.39** | **86.62** | **87.98** | 2870 | -| all fields (macro avg.) | 89.41 | 86.62 | 88 | 2870 | +| label | precision | recall | f1 | support | +|-----------------------------|-----------|-----------|-----------|---------| +| authors | 80 | 77.74 | 78.86 | 957 | +| first_author | 91.83 | 89.33 | 90.56 | 956 | +| title | 96.42 | 92.79 | 94.57 | 957 | +| | | | | | +| **all fields (micro avg.)** | **89.39** | **86.62** | **87.98** | 2870 | +| all fields (macro avg.) | 89.41 | 86.62 | 88 | 2870 | @@ -37,14 +37,14 @@ Evaluation on 957 random PDF files out of 982 PDF (ratio 1.0). **Field-level results** -| label | precision | recall | f1 | support | -|--- |--- |--- |--- |--- | -| authors | 92.8 | 90.18 | 91.47 | 957 | -| first_author | 92.26 | 89.75 | 90.99 | 956 | -| title | 98.05 | 94.36 | 96.17 | 957 | -| | | | | | -| **all fields (micro avg.)** | **94.35** | **91.43** | **92.87** | 2870 | -| all fields (macro avg.) | 94.37 | 91.43 | 92.87 | 2870 | +| label | precision | recall | f1 | support | +|-----------------------------|-----------|-----------|-----------|---------| +| authors | 92.8 | 90.18 | 91.47 | 957 | +| first_author | 92.26 | 89.75 | 90.99 | 956 | +| title | 98.05 | 94.36 | 96.17 | 957 | +| | | | | | +| **all fields (micro avg.)** | **94.35** | **91.43** | **92.87** | 2870 | +| all fields (macro avg.) | 94.37 | 91.43 | 92.87 | 2870 | @@ -52,14 +52,14 @@ Evaluation on 957 random PDF files out of 982 PDF (ratio 1.0). **Field-level results** -| label | precision | recall | f1 | support | -|--- |--- |--- |--- |--- | -| authors | 85.59 | 83.18 | 84.37 | 957 | -| first_author | 91.83 | 89.33 | 90.56 | 956 | -| title | 97.94 | 94.25 | 96.06 | 957 | -| | | | | | -| **all fields (micro avg.)** | **91.77** | **88.92** | **90.32** | 2870 | -| all fields (macro avg.) | 91.79 | 88.92 | 90.33 | 2870 | +| label | precision | recall | f1 | support | +|-----------------------------|-----------|-----------|-----------|---------| +| authors | 85.59 | 83.18 | 84.37 | 957 | +| first_author | 91.83 | 89.33 | 90.56 | 956 | +| title | 97.94 | 94.25 | 96.06 | 957 | +| | | | | | +| **all fields (micro avg.)** | **91.77** | **88.92** | **90.32** | 2870 | +| all fields (macro avg.) | 91.79 | 88.92 | 90.33 | 2870 | #### Instance-level results diff --git a/doc/benchmarks/flavors/article_light/benchmaking-plos.md b/doc/benchmarks/flavors/article_light/benchmaking-plos.md index a108a29f41..c76226c0e2 100644 --- a/doc/benchmarks/flavors/article_light/benchmaking-plos.md +++ b/doc/benchmarks/flavors/article_light/benchmaking-plos.md @@ -7,14 +7,14 @@ Evaluation on 1000 random PDF files out of 998 PDF (ratio 1.0). **Field-level results** -| label | precision | recall | f1 | support | -|--- |--- |--- |--- |--- | -| authors | 98.97 | 99.07 | 99.02 | 969 | -| first_author | 99.28 | 99.38 | 99.33 | 969 | -| title | 95.77 | 95.1 | 95.43 | 1000 | -| | | | | | -| **all fields (micro avg.)** | **97.99** | **97.82** | **97.9** | 2938 | -| all fields (macro avg.) | 98.01 | 97.85 | 97.93 | 2938 | +| label | precision | recall | f1 | support | +|-----------------------------|-----------|-----------|----------|---------| +| authors | 98.97 | 99.07 | 99.02 | 969 | +| first_author | 99.28 | 99.38 | 99.33 | 969 | +| title | 95.77 | 95.1 | 95.43 | 1000 | +| | | | | | +| **all fields (micro avg.)** | **97.99** | **97.82** | **97.9** | 2938 | +| all fields (macro avg.) | 98.01 | 97.85 | 97.93 | 2938 | @@ -22,14 +22,14 @@ Evaluation on 1000 random PDF files out of 998 PDF (ratio 1.0). **Field-level results** -| label | precision | recall | f1 | support | -|--- |--- |--- |--- |--- | -| authors | 98.97 | 99.07 | 99.02 | 969 | -| first_author | 99.28 | 99.38 | 99.33 | 969 | -| title | 99.3 | 98.6 | 98.95 | 1000 | -| | | | | | -| **all fields (micro avg.)** | **99.18** | **99.01** | **99.1** | 2938 | -| all fields (macro avg.) | 99.18 | 99.02 | 99.1 | 2938 | +| label | precision | recall | f1 | support | +|-----------------------------|-----------|-----------|----------|---------| +| authors | 98.97 | 99.07 | 99.02 | 969 | +| first_author | 99.28 | 99.38 | 99.33 | 969 | +| title | 99.3 | 98.6 | 98.95 | 1000 | +| | | | | | +| **all fields (micro avg.)** | **99.18** | **99.01** | **99.1** | 2938 | +| all fields (macro avg.) | 99.18 | 99.02 | 99.1 | 2938 | @@ -37,14 +37,14 @@ Evaluation on 1000 random PDF files out of 998 PDF (ratio 1.0). **Field-level results** -| label | precision | recall | f1 | support | -|--- |--- |--- |--- |--- | -| authors | 99.28 | 99.38 | 99.33 | 969 | -| first_author | 99.38 | 99.48 | 99.43 | 969 | -| title | 99.7 | 99 | 99.35 | 1000 | -| | | | | | -| **all fields (micro avg.)** | **99.45** | **99.29** | **99.37** | 2938 | -| all fields (macro avg.) | 99.45 | 99.29 | 99.37 | 2938 | +| label | precision | recall | f1 | support | +|-----------------------------|-----------|-----------|------------|---------| +| authors | 99.28 | 99.38 | 99.33 | 969 | +| first_author | 99.38 | 99.48 | 99.43 | 969 | +| title | 99.7 | 99 | 99.35 | 1000 | +| | | | | | +| **all fields (micro avg.)** | **99.45** | **99.29** | **99.37** | 2938 | +| all fields (macro avg.) | 99.45 | 99.29 | 99.37 | 2938 | @@ -52,14 +52,14 @@ Evaluation on 1000 random PDF files out of 998 PDF (ratio 1.0). **Field-level results** -| label | precision | recall | f1 | support | -|--- |--- |--- |--- |--- | -| authors | 99.18 | 99.28 | 99.23 | 969 | -| first_author | 99.28 | 99.38 | 99.33 | 969 | -| title | 99.5 | 98.8 | 99.15 | 1000 | -| | | | | | -| **all fields (micro avg.)** | **99.32** | **99.15** | **99.23** | 2938 | -| all fields (macro avg.) | 99.32 | 99.15 | 99.23 | 2938 | +| label | precision | recall | f1 | support | +|-----------------------------|-----------|-----------|-----------|---------| +| authors | 99.18 | 99.28 | 99.23 | 969 | +| first_author | 99.28 | 99.38 | 99.33 | 969 | +| title | 99.5 | 98.8 | 99.15 | 1000 | +| | | | | | +| **all fields (micro avg.)** | **99.32** | **99.15** | **99.23** | 2938 | +| all fields (macro avg.) | 99.32 | 99.15 | 99.23 | 2938 | #### Instance-level results diff --git a/doc/benchmarks/flavors/article_light/benchmaking-pmc.md b/doc/benchmarks/flavors/article_light/benchmaking-pmc.md index 96c13a12a5..ea0603513e 100644 --- a/doc/benchmarks/flavors/article_light/benchmaking-pmc.md +++ b/doc/benchmarks/flavors/article_light/benchmaking-pmc.md @@ -7,14 +7,14 @@ Evaluation on 1943 random PDF files out of 1941 PDF (ratio 1.0). **Field-level results** -| label | precision | recall | f1 | support | -|--- |--- |--- |--- |--- | -| authors | 92.2 | 91.91 | 92.05 | 1941 | -| first_author | 96.28 | 95.98 | 96.13 | 1941 | -| title | 84.33 | 83.38 | 83.85 | 1943 | -| | | | | | -| **all fields (micro avg.)** | **90.95** | **90.42** | **90.69** | 5825 | -| all fields (macro avg.) | 90.94 | 90.42 | 90.68 | 5825 | +| label | precision | recall | f1 | support | +|-----------------------------|-----------|-----------|-----------|---------| +| authors | 92.2 | 91.91 | 92.05 | 1941 | +| first_author | 96.28 | 95.98 | 96.13 | 1941 | +| title | 84.33 | 83.38 | 83.85 | 1943 | +| | | | | | +| **all fields (micro avg.)** | **90.95** | **90.42** | **90.69** | 5825 | +| all fields (macro avg.) | 90.94 | 90.42 | 90.68 | 5825 | @@ -22,14 +22,14 @@ Evaluation on 1943 random PDF files out of 1941 PDF (ratio 1.0). **Field-level results** -| label | precision | recall | f1 | support | -|--- |--- |--- |--- |--- | -| authors | 94.11 | 93.82 | 93.96 | 1941 | -| first_author | 96.64 | 96.34 | 96.49 | 1941 | -| title | 92.04 | 90.99 | 91.51 | 1943 | -| | | | | | -| **all fields (micro avg.)** | **94.27** | **93.72** | **93.99** | 5825 | -| all fields (macro avg.) | 94.26 | 93.72 | 93.99 | 5825 | +| label | precision | recall | f1 | support | +|-----------------------------|-----------|-----------|-----------|---------| +| authors | 94.11 | 93.82 | 93.96 | 1941 | +| first_author | 96.64 | 96.34 | 96.49 | 1941 | +| title | 92.04 | 90.99 | 91.51 | 1943 | +| | | | | | +| **all fields (micro avg.)** | **94.27** | **93.72** | **93.99** | 5825 | +| all fields (macro avg.) | 94.26 | 93.72 | 93.99 | 5825 | @@ -37,14 +37,14 @@ Evaluation on 1943 random PDF files out of 1941 PDF (ratio 1.0). **Field-level results** -| label | precision | recall | f1 | support | -|--- |--- |--- |--- |--- | -| authors | 96.28 | 95.98 | 96.13 | 1941 | -| first_author | 96.95 | 96.65 | 96.8 | 1941 | -| title | 98.18 | 97.07 | 97.62 | 1943 | -| | | | | | -| **all fields (micro avg.)** | **97.13** | **96.57** | **96.85** | 5825 | -| all fields (macro avg.) | 97.14 | 96.57 | 96.85 | 5825 | +| label | precision | recall | f1 | support | +|-----------------------------|-----------|-----------|-----------|---------| +| authors | 96.28 | 95.98 | 96.13 | 1941 | +| first_author | 96.95 | 96.65 | 96.8 | 1941 | +| title | 98.18 | 97.07 | 97.62 | 1943 | +| | | | | | +| **all fields (micro avg.)** | **97.13** | **96.57** | **96.85** | 5825 | +| all fields (macro avg.) | 97.14 | 96.57 | 96.85 | 5825 | @@ -52,14 +52,14 @@ Evaluation on 1943 random PDF files out of 1941 PDF (ratio 1.0). **Field-level results** -| label | precision | recall | f1 | support | -|--- |--- |--- |--- |--- | -| authors | 95.3 | 95 | 95.15 | 1941 | -| first_author | 96.28 | 95.98 | 96.13 | 1941 | -| title | 96.2 | 95.11 | 95.65 | 1943 | -| | | | | | -| **all fields (micro avg.)** | **95.92** | **95.36** | **95.64** | 5825 | -| all fields (macro avg.) | 95.93 | 95.36 | 95.64 | 5825 | +| label | precision | recall | f1 | support | +|-----------------------------|-----------|-----------|-----------|---------| +| authors | 95.3 | 95 | 95.15 | 1941 | +| first_author | 96.28 | 95.98 | 96.13 | 1941 | +| title | 96.2 | 95.11 | 95.65 | 1943 | +| | | | | | +| **all fields (micro avg.)** | **95.92** | **95.36** | **95.64** | 5825 | +| all fields (macro avg.) | 95.93 | 95.36 | 95.64 | 5825 | #### Instance-level results diff --git a/doc/benchmarks/flavors/article_light_ref/benchmaking-biorxiv.md b/doc/benchmarks/flavors/article_light_ref/benchmaking-biorxiv.md index 1c78875302..9032eeebc4 100644 --- a/doc/benchmarks/flavors/article_light_ref/benchmaking-biorxiv.md +++ b/doc/benchmarks/flavors/article_light_ref/benchmaking-biorxiv.md @@ -1,5 +1,5 @@ -## Header metadata +## Header metadata Evaluation on 2000 random PDF files out of 1998 PDF (ratio 1.0). @@ -7,60 +7,53 @@ Evaluation on 2000 random PDF files out of 1998 PDF (ratio 1.0). **Field-level results** -| label | precision | recall | f1 | support | -|--- |--- |--- |--- |--- | -| authors | 83.38 | 81.79 | 82.58 | 1999 | -| first_author | 96.58 | 94.84 | 95.7 | 1997 | -| title | 78.19 | 73.85 | 75.96 | 2000 | -| | | | | | -| **all fields (micro avg.)** | **86.15** | **83.49** | **84.8** | 5996 | -| all fields (macro avg.) | 86.05 | 83.49 | 84.75 | 5996 | - - +| label | precision | recall | f1 | support | +|-----------------------------|-----------|-----------|----------|---------| +| authors | 83.38 | 81.79 | 82.58 | 1999 | +| first_author | 96.58 | 94.84 | 95.7 | 1997 | +| title | 78.19 | 73.85 | 75.96 | 2000 | +| | | | | | +| **all fields (micro avg.)** | **86.15** | **83.49** | **84.8** | 5996 | +| all fields (macro avg.) | 86.05 | 83.49 | 84.75 | 5996 | #### Soft Matching (ignoring punctuation, case and space characters mismatches) **Field-level results** -| label | precision | recall | f1 | support | -|--- |--- |--- |--- |--- | -| authors | 83.94 | 82.34 | 83.13 | 1999 | -| first_author | 96.89 | 95.14 | 96.01 | 1997 | -| title | 80.57 | 76.1 | 78.27 | 2000 | -| | | | | | -| **all fields (micro avg.)** | **87.21** | **84.52** | **85.85** | 5996 | -| all fields (macro avg.) | 87.13 | 84.53 | 85.8 | 5996 | - - +| label | precision | recall | f1 | support | +|-----------------------------|-----------|-----------|-----------|---------| +| authors | 83.94 | 82.34 | 83.13 | 1999 | +| first_author | 96.89 | 95.14 | 96.01 | 1997 | +| title | 80.57 | 76.1 | 78.27 | 2000 | +| | | | | | +| **all fields (micro avg.)** | **87.21** | **84.52** | **85.85** | 5996 | +| all fields (macro avg.) | 87.13 | 84.53 | 85.8 | 5996 | #### Levenshtein Matching (Minimum Levenshtein distance at 0.8) **Field-level results** -| label | precision | recall | f1 | support | -|--- |--- |--- |--- |--- | -| authors | 92.04 | 90.3 | 91.16 | 1999 | -| first_author | 97.04 | 95.29 | 96.16 | 1997 | -| title | 92.11 | 87 | 89.48 | 2000 | -| | | | | | -| **all fields (micro avg.)** | **93.75** | **90.86** | **92.28** | 5996 | -| all fields (macro avg.) | 93.73 | 90.86 | 92.27 | 5996 | - - +| label | precision | recall | f1 | support | +|-----------------------------|-----------|-----------|-----------|---------| +| authors | 92.04 | 90.3 | 91.16 | 1999 | +| first_author | 97.04 | 95.29 | 96.16 | 1997 | +| title | 92.11 | 87 | 89.48 | 2000 | +| | | | | | +| **all fields (micro avg.)** | **93.75** | **90.86** | **92.28** | 5996 | +| all fields (macro avg.) | 93.73 | 90.86 | 92.27 | 5996 | #### Ratcliff/Obershelp Matching (Minimum Ratcliff/Obershelp similarity at 0.95) **Field-level results** -| label | precision | recall | f1 | support | -|--- |--- |--- |--- |--- | -| authors | 88.02 | 86.34 | 87.17 | 1999 | -| first_author | 96.58 | 94.84 | 95.7 | 1997 | -| title | 88.35 | 83.45 | 85.83 | 2000 | -| | | | | | -| **all fields (micro avg.)** | **91.02** | **88.21** | **89.59** | 5996 | -| all fields (macro avg.) | 90.98 | 88.21 | 89.57 | 5996 | - +| label | precision | recall | f1 | support | +|-----------------------------|-----------|-----------|-----------|---------| +| authors | 88.02 | 86.34 | 87.17 | 1999 | +| first_author | 96.58 | 94.84 | 95.7 | 1997 | +| title | 88.35 | 83.45 | 85.83 | 2000 | +| | | | | | +| **all fields (micro avg.)** | **91.02** | **88.21** | **89.59** | 5996 | +| all fields (macro avg.) | 90.98 | 88.21 | 89.57 | 5996 | #### Instance-level results @@ -77,8 +70,7 @@ Instance-level recall: 81.35 (Levenshtein) Instance-level recall: 75.35 (RatcliffObershelp) ``` - -## Citation metadata +## Citation metadata Evaluation on 2000 random PDF files out of 1998 PDF (ratio 1.0). @@ -86,92 +78,85 @@ Evaluation on 2000 random PDF files out of 1998 PDF (ratio 1.0). **Field-level results** -| label | precision | recall | f1 | support | -|--- |--- |--- |--- |--- | -| authors | 88.07 | 82.47 | 85.18 | 97183 | -| date | 91.66 | 85.55 | 88.5 | 97630 | -| doi | 70.88 | 82.18 | 76.11 | 16894 | -| first_author | 94.98 | 88.85 | 91.82 | 97183 | -| inTitle | 82.76 | 78.67 | 80.66 | 96430 | -| issue | 94.32 | 91.01 | 92.64 | 30312 | -| page | 94.93 | 77.66 | 85.43 | 88597 | -| pmcid | 65.91 | 82.65 | 73.34 | 807 | -| pmid | 69.03 | 81.46 | 74.73 | 2093 | -| title | 84.81 | 82.78 | 83.78 | 92463 | -| volume | 96.19 | 94.39 | 95.28 | 87709 | -| | | | | | -| **all fields (micro avg.)** | **89.79** | **84.52** | **87.08** | 707301 | -| all fields (macro avg.) | 84.87 | 84.33 | 84.31 | 707301 | - - +| label | precision | recall | f1 | support | +|-----------------------------|-----------|-----------|-----------|---------| +| authors | 88.07 | 82.47 | 85.18 | 97183 | +| date | 91.66 | 85.55 | 88.5 | 97630 | +| doi | 70.88 | 82.18 | 76.11 | 16894 | +| first_author | 94.98 | 88.85 | 91.82 | 97183 | +| inTitle | 82.76 | 78.67 | 80.66 | 96430 | +| issue | 94.32 | 91.01 | 92.64 | 30312 | +| page | 94.93 | 77.66 | 85.43 | 88597 | +| pmcid | 65.91 | 82.65 | 73.34 | 807 | +| pmid | 69.03 | 81.46 | 74.73 | 2093 | +| title | 84.81 | 82.78 | 83.78 | 92463 | +| volume | 96.19 | 94.39 | 95.28 | 87709 | +| | | | | | +| **all fields (micro avg.)** | **89.79** | **84.52** | **87.08** | 707301 | +| all fields (macro avg.) | 84.87 | 84.33 | 84.31 | 707301 | #### Soft Matching (ignoring punctuation, case and space characters mismatches) **Field-level results** -| label | precision | recall | f1 | support | -|--- |--- |--- |--- |--- | -| authors | 89.23 | 83.55 | 86.29 | 97183 | -| date | 91.66 | 85.55 | 88.5 | 97630 | -| doi | 75.35 | 87.37 | 80.92 | 16894 | -| first_author | 95.41 | 89.26 | 92.23 | 97183 | -| inTitle | 92.29 | 87.73 | 89.95 | 96430 | -| issue | 94.32 | 91.01 | 92.64 | 30312 | -| page | 94.93 | 77.66 | 85.43 | 88597 | -| pmcid | 75.3 | 94.42 | 83.78 | 807 | -| pmid | 73.56 | 86.81 | 79.64 | 2093 | -| title | 93.15 | 90.92 | 92.02 | 92463 | -| volume | 96.19 | 94.39 | 95.28 | 87709 | -| | | | | | -| **all fields (micro avg.)** | **92.62** | **87.18** | **89.82** | 707301 | -| all fields (macro avg.) | 88.31 | 88.06 | 87.88 | 707301 | - - +| label | precision | recall | f1 | support | +|-----------------------------|-----------|-----------|-----------|---------| +| authors | 89.23 | 83.55 | 86.29 | 97183 | +| date | 91.66 | 85.55 | 88.5 | 97630 | +| doi | 75.35 | 87.37 | 80.92 | 16894 | +| first_author | 95.41 | 89.26 | 92.23 | 97183 | +| inTitle | 92.29 | 87.73 | 89.95 | 96430 | +| issue | 94.32 | 91.01 | 92.64 | 30312 | +| page | 94.93 | 77.66 | 85.43 | 88597 | +| pmcid | 75.3 | 94.42 | 83.78 | 807 | +| pmid | 73.56 | 86.81 | 79.64 | 2093 | +| title | 93.15 | 90.92 | 92.02 | 92463 | +| volume | 96.19 | 94.39 | 95.28 | 87709 | +| | | | | | +| **all fields (micro avg.)** | **92.62** | **87.18** | **89.82** | 707301 | +| all fields (macro avg.) | 88.31 | 88.06 | 87.88 | 707301 | #### Levenshtein Matching (Minimum Levenshtein distance at 0.8) **Field-level results** -| label | precision | recall | f1 | support | -|--- |--- |--- |--- |--- | -| authors | 94.48 | 88.47 | 91.38 | 97183 | -| date | 91.66 | 85.55 | 88.5 | 97630 | -| doi | 77.54 | 89.91 | 83.27 | 16894 | -| first_author | 95.56 | 89.39 | 92.37 | 97183 | -| inTitle | 93.24 | 88.64 | 90.88 | 96430 | -| issue | 94.32 | 91.01 | 92.64 | 30312 | -| page | 94.93 | 77.66 | 85.43 | 88597 | -| pmcid | 75.3 | 94.42 | 83.78 | 807 | -| pmid | 73.56 | 86.81 | 79.64 | 2093 | -| title | 95.97 | 93.67 | 94.81 | 92463 | -| volume | 96.19 | 94.39 | 95.28 | 87709 | -| | | | | | -| **all fields (micro avg.)** | **93.93** | **88.42** | **91.09** | 707301 | -| all fields (macro avg.) | 89.34 | 89.08 | 88.91 | 707301 | - - +| label | precision | recall | f1 | support | +|-----------------------------|-----------|-----------|-----------|---------| +| authors | 94.48 | 88.47 | 91.38 | 97183 | +| date | 91.66 | 85.55 | 88.5 | 97630 | +| doi | 77.54 | 89.91 | 83.27 | 16894 | +| first_author | 95.56 | 89.39 | 92.37 | 97183 | +| inTitle | 93.24 | 88.64 | 90.88 | 96430 | +| issue | 94.32 | 91.01 | 92.64 | 30312 | +| page | 94.93 | 77.66 | 85.43 | 88597 | +| pmcid | 75.3 | 94.42 | 83.78 | 807 | +| pmid | 73.56 | 86.81 | 79.64 | 2093 | +| title | 95.97 | 93.67 | 94.81 | 92463 | +| volume | 96.19 | 94.39 | 95.28 | 87709 | +| | | | | | +| **all fields (micro avg.)** | **93.93** | **88.42** | **91.09** | 707301 | +| all fields (macro avg.) | 89.34 | 89.08 | 88.91 | 707301 | #### Ratcliff/Obershelp Matching (Minimum Ratcliff/Obershelp similarity at 0.95) **Field-level results** -| label | precision | recall | f1 | support | -|--- |--- |--- |--- |--- | -| authors | 91.44 | 85.62 | 88.43 | 97183 | -| date | 91.66 | 85.55 | 88.5 | 97630 | -| doi | 76.04 | 88.17 | 81.66 | 16894 | -| first_author | 95.03 | 88.9 | 91.86 | 97183 | -| inTitle | 90.97 | 86.48 | 88.67 | 96430 | -| issue | 94.32 | 91.01 | 92.64 | 30312 | -| page | 94.93 | 77.66 | 85.43 | 88597 | -| pmcid | 65.91 | 82.65 | 73.34 | 807 | -| pmid | 69.03 | 81.46 | 74.73 | 2093 | -| title | 95.24 | 92.96 | 94.09 | 92463 | -| volume | 96.19 | 94.39 | 95.28 | 87709 | -| | | | | | -| **all fields (micro avg.)** | **92.96** | **87.5** | **90.15** | 707301 | -| all fields (macro avg.) | 87.34 | 86.8 | 86.78 | 707301 | - +| label | precision | recall | f1 | support | +|-----------------------------|-----------|----------|-----------|---------| +| authors | 91.44 | 85.62 | 88.43 | 97183 | +| date | 91.66 | 85.55 | 88.5 | 97630 | +| doi | 76.04 | 88.17 | 81.66 | 16894 | +| first_author | 95.03 | 88.9 | 91.86 | 97183 | +| inTitle | 90.97 | 86.48 | 88.67 | 96430 | +| issue | 94.32 | 91.01 | 92.64 | 30312 | +| page | 94.93 | 77.66 | 85.43 | 88597 | +| pmcid | 65.91 | 82.65 | 73.34 | 807 | +| pmid | 69.03 | 81.46 | 74.73 | 2093 | +| title | 95.24 | 92.96 | 94.09 | 92463 | +| volume | 96.19 | 94.39 | 95.28 | 87709 | +| | | | | | +| **all fields (micro avg.)** | **92.96** | **87.5** | **90.15** | 707301 | +| all fields (macro avg.) | 87.34 | 86.8 | 86.78 | 707301 | #### Instance-level results @@ -209,8 +194,8 @@ Matching 4 : 2084 Total matches : 89410 ``` - #### Citation context resolution + ``` Total expected references: 98797 - 49.4 references per article diff --git a/doc/benchmarks/flavors/article_light_ref/benchmaking-elife.md b/doc/benchmarks/flavors/article_light_ref/benchmaking-elife.md index 933c51d365..af40f2ead8 100644 --- a/doc/benchmarks/flavors/article_light_ref/benchmaking-elife.md +++ b/doc/benchmarks/flavors/article_light_ref/benchmaking-elife.md @@ -1,5 +1,5 @@ -## Header metadata +## Header metadata Evaluation on 984 random PDF files out of 982 PDF (ratio 1.0). @@ -7,60 +7,53 @@ Evaluation on 984 random PDF files out of 982 PDF (ratio 1.0). **Field-level results** -| label | precision | recall | f1 | support | -|--- |--- |--- |--- |--- | -| authors | 80.42 | 78.13 | 79.26 | 983 | -| first_author | 91.62 | 89.1 | 90.35 | 982 | -| title | 89.24 | 85.98 | 87.58 | 984 | -| | | | | | -| **all fields (micro avg.)** | **87.09** | **84.4** | **85.72** | 2949 | -| all fields (macro avg.) | 87.09 | 84.4 | 85.73 | 2949 | - - +| label | precision | recall | f1 | support | +|-----------------------------|-----------|----------|-----------|---------| +| authors | 80.42 | 78.13 | 79.26 | 983 | +| first_author | 91.62 | 89.1 | 90.35 | 982 | +| title | 89.24 | 85.98 | 87.58 | 984 | +| | | | | | +| **all fields (micro avg.)** | **87.09** | **84.4** | **85.72** | 2949 | +| all fields (macro avg.) | 87.09 | 84.4 | 85.73 | 2949 | #### Soft Matching (ignoring punctuation, case and space characters mismatches) **Field-level results** -| label | precision | recall | f1 | support | -|--- |--- |--- |--- |--- | -| authors | 80.73 | 78.43 | 79.57 | 983 | -| first_author | 91.62 | 89.1 | 90.35 | 982 | -| title | 96.1 | 92.58 | 94.31 | 984 | -| | | | | | -| **all fields (micro avg.)** | **89.47** | **86.71** | **88.07** | 2949 | -| all fields (macro avg.) | 89.48 | 86.71 | 88.07 | 2949 | - - +| label | precision | recall | f1 | support | +|-----------------------------|-----------|-----------|-----------|---------| +| authors | 80.73 | 78.43 | 79.57 | 983 | +| first_author | 91.62 | 89.1 | 90.35 | 982 | +| title | 96.1 | 92.58 | 94.31 | 984 | +| | | | | | +| **all fields (micro avg.)** | **89.47** | **86.71** | **88.07** | 2949 | +| all fields (macro avg.) | 89.48 | 86.71 | 88.07 | 2949 | #### Levenshtein Matching (Minimum Levenshtein distance at 0.8) **Field-level results** -| label | precision | recall | f1 | support | -|--- |--- |--- |--- |--- | -| authors | 92.77 | 90.13 | 91.43 | 983 | -| first_author | 91.94 | 89.41 | 90.66 | 982 | -| title | 97.57 | 94 | 95.76 | 984 | -| | | | | | -| **all fields (micro avg.)** | **94.09** | **91.18** | **92.61** | 2949 | -| all fields (macro avg.) | 94.1 | 91.18 | 92.62 | 2949 | - - +| label | precision | recall | f1 | support | +|-----------------------------|-----------|-----------|-----------|---------| +| authors | 92.77 | 90.13 | 91.43 | 983 | +| first_author | 91.94 | 89.41 | 90.66 | 982 | +| title | 97.57 | 94 | 95.76 | 984 | +| | | | | | +| **all fields (micro avg.)** | **94.09** | **91.18** | **92.61** | 2949 | +| all fields (macro avg.) | 94.1 | 91.18 | 92.62 | 2949 | #### Ratcliff/Obershelp Matching (Minimum Ratcliff/Obershelp similarity at 0.95) **Field-level results** -| label | precision | recall | f1 | support | -|--- |--- |--- |--- |--- | -| authors | 86.07 | 83.62 | 84.83 | 983 | -| first_author | 91.62 | 89.1 | 90.35 | 982 | -| title | 97.57 | 94 | 95.76 | 984 | -| | | | | | -| **all fields (micro avg.)** | **91.74** | **88.91** | **90.3** | 2949 | -| all fields (macro avg.) | 91.76 | 88.91 | 90.31 | 2949 | - +| label | precision | recall | f1 | support | +|-----------------------------|-----------|-----------|----------|---------| +| authors | 86.07 | 83.62 | 84.83 | 983 | +| first_author | 91.62 | 89.1 | 90.35 | 982 | +| title | 97.57 | 94 | 95.76 | 984 | +| | | | | | +| **all fields (micro avg.)** | **91.74** | **88.91** | **90.3** | 2949 | +| all fields (macro avg.) | 91.76 | 88.91 | 90.31 | 2949 | #### Instance-level results @@ -77,8 +70,7 @@ Instance-level recall: 85.37 (Levenshtein) Instance-level recall: 81.2 (RatcliffObershelp) ``` - -## Citation metadata +## Citation metadata Evaluation on 984 random PDF files out of 982 PDF (ratio 1.0). @@ -86,80 +78,73 @@ Evaluation on 984 random PDF files out of 982 PDF (ratio 1.0). **Field-level results** -| label | precision | recall | f1 | support | -|--- |--- |--- |--- |--- | -| authors | 79.41 | 78.03 | 78.71 | 63265 | -| date | 95.87 | 93.79 | 94.82 | 63662 | -| first_author | 94.8 | 93.12 | 93.95 | 63265 | -| inTitle | 95.79 | 94.52 | 95.15 | 63213 | -| issue | 1.98 | 75 | 3.85 | 16 | -| page | 96.25 | 95.04 | 95.65 | 53375 | -| title | 90.25 | 90.53 | 90.39 | 62044 | -| volume | 97.89 | 97.99 | 97.94 | 61049 | -| | | | | | -| **all fields (micro avg.)** | **92.68** | **91.76** | **92.22** | 429889 | -| all fields (macro avg.) | 81.53 | 89.75 | 81.31 | 429889 | - - +| label | precision | recall | f1 | support | +|-----------------------------|-----------|-----------|-----------|---------| +| authors | 79.41 | 78.03 | 78.71 | 63265 | +| date | 95.87 | 93.79 | 94.82 | 63662 | +| first_author | 94.8 | 93.12 | 93.95 | 63265 | +| inTitle | 95.79 | 94.52 | 95.15 | 63213 | +| issue | 1.98 | 75 | 3.85 | 16 | +| page | 96.25 | 95.04 | 95.65 | 53375 | +| title | 90.25 | 90.53 | 90.39 | 62044 | +| volume | 97.89 | 97.99 | 97.94 | 61049 | +| | | | | | +| **all fields (micro avg.)** | **92.68** | **91.76** | **92.22** | 429889 | +| all fields (macro avg.) | 81.53 | 89.75 | 81.31 | 429889 | #### Soft Matching (ignoring punctuation, case and space characters mismatches) **Field-level results** -| label | precision | recall | f1 | support | -|--- |--- |--- |--- |--- | -| authors | 79.54 | 78.17 | 78.85 | 63265 | -| date | 95.87 | 93.79 | 94.82 | 63662 | -| first_author | 94.88 | 93.2 | 94.03 | 63265 | -| inTitle | 96.27 | 94.99 | 95.63 | 63213 | -| issue | 1.98 | 75 | 3.85 | 16 | -| page | 96.25 | 95.04 | 95.65 | 53375 | -| title | 95.9 | 96.2 | 96.05 | 62044 | -| volume | 97.89 | 97.99 | 97.94 | 61049 | -| | | | | | -| **all fields (micro avg.)** | **93.61** | **92.68** | **93.14** | 429889 | -| all fields (macro avg.) | 82.32 | 90.55 | 82.1 | 429889 | - - +| label | precision | recall | f1 | support | +|-----------------------------|-----------|-----------|-----------|---------| +| authors | 79.54 | 78.17 | 78.85 | 63265 | +| date | 95.87 | 93.79 | 94.82 | 63662 | +| first_author | 94.88 | 93.2 | 94.03 | 63265 | +| inTitle | 96.27 | 94.99 | 95.63 | 63213 | +| issue | 1.98 | 75 | 3.85 | 16 | +| page | 96.25 | 95.04 | 95.65 | 53375 | +| title | 95.9 | 96.2 | 96.05 | 62044 | +| volume | 97.89 | 97.99 | 97.94 | 61049 | +| | | | | | +| **all fields (micro avg.)** | **93.61** | **92.68** | **93.14** | 429889 | +| all fields (macro avg.) | 82.32 | 90.55 | 82.1 | 429889 | #### Levenshtein Matching (Minimum Levenshtein distance at 0.8) **Field-level results** -| label | precision | recall | f1 | support | -|--- |--- |--- |--- |--- | -| authors | 93.29 | 91.68 | 92.48 | 63265 | -| date | 95.87 | 93.79 | 94.82 | 63662 | -| first_author | 95.33 | 93.64 | 94.48 | 63265 | -| inTitle | 96.59 | 95.31 | 95.95 | 63213 | -| issue | 1.98 | 75 | 3.85 | 16 | -| page | 96.25 | 95.04 | 95.65 | 53375 | -| title | 97.63 | 97.94 | 97.78 | 62044 | -| volume | 97.89 | 97.99 | 97.94 | 61049 | -| | | | | | -| **all fields (micro avg.)** | **95.98** | **95.03** | **95.5** | 429889 | -| all fields (macro avg.) | 84.36 | 92.55 | 84.12 | 429889 | - - +| label | precision | recall | f1 | support | +|-----------------------------|-----------|-----------|----------|---------| +| authors | 93.29 | 91.68 | 92.48 | 63265 | +| date | 95.87 | 93.79 | 94.82 | 63662 | +| first_author | 95.33 | 93.64 | 94.48 | 63265 | +| inTitle | 96.59 | 95.31 | 95.95 | 63213 | +| issue | 1.98 | 75 | 3.85 | 16 | +| page | 96.25 | 95.04 | 95.65 | 53375 | +| title | 97.63 | 97.94 | 97.78 | 62044 | +| volume | 97.89 | 97.99 | 97.94 | 61049 | +| | | | | | +| **all fields (micro avg.)** | **95.98** | **95.03** | **95.5** | 429889 | +| all fields (macro avg.) | 84.36 | 92.55 | 84.12 | 429889 | #### Ratcliff/Obershelp Matching (Minimum Ratcliff/Obershelp similarity at 0.95) **Field-level results** -| label | precision | recall | f1 | support | -|--- |--- |--- |--- |--- | -| authors | 86.72 | 85.22 | 85.96 | 63265 | -| date | 95.87 | 93.79 | 94.82 | 63662 | -| first_author | 94.82 | 93.13 | 93.97 | 63265 | -| inTitle | 96.27 | 95 | 95.63 | 63213 | -| issue | 1.98 | 75 | 3.85 | 16 | -| page | 96.25 | 95.04 | 95.65 | 53375 | -| title | 97.49 | 97.79 | 97.64 | 62044 | -| volume | 97.89 | 97.99 | 97.94 | 61049 | -| | | | | | -| **all fields (micro avg.)** | **94.88** | **93.94** | **94.41** | 429889 | -| all fields (macro avg.) | 83.41 | 91.62 | 83.18 | 429889 | - +| label | precision | recall | f1 | support | +|-----------------------------|-----------|-----------|-----------|---------| +| authors | 86.72 | 85.22 | 85.96 | 63265 | +| date | 95.87 | 93.79 | 94.82 | 63662 | +| first_author | 94.82 | 93.13 | 93.97 | 63265 | +| inTitle | 96.27 | 95 | 95.63 | 63213 | +| issue | 1.98 | 75 | 3.85 | 16 | +| page | 96.25 | 95.04 | 95.65 | 53375 | +| title | 97.49 | 97.79 | 97.64 | 62044 | +| volume | 97.89 | 97.99 | 97.94 | 61049 | +| | | | | | +| **all fields (micro avg.)** | **94.88** | **93.94** | **94.41** | 429889 | +| all fields (macro avg.) | 83.41 | 91.62 | 83.18 | 429889 | #### Instance-level results @@ -197,8 +182,8 @@ Matching 4 : 365 Total matches : 61101 ``` - #### Citation context resolution + ``` Total expected references: 63664 - 64.7 references per article diff --git a/doc/benchmarks/flavors/article_light_ref/benchmaking-plos.md b/doc/benchmarks/flavors/article_light_ref/benchmaking-plos.md index be9662a860..63d126928e 100644 --- a/doc/benchmarks/flavors/article_light_ref/benchmaking-plos.md +++ b/doc/benchmarks/flavors/article_light_ref/benchmaking-plos.md @@ -1,5 +1,5 @@ -## Header metadata +## Header metadata Evaluation on 1000 random PDF files out of 998 PDF (ratio 1.0). @@ -7,60 +7,53 @@ Evaluation on 1000 random PDF files out of 998 PDF (ratio 1.0). **Field-level results** -| label | precision | recall | f1 | support | -|--- |--- |--- |--- |--- | -| authors | 99.07 | 99.07 | 99.07 | 969 | -| first_author | 99.38 | 99.38 | 99.38 | 969 | -| title | 95.76 | 94.8 | 95.28 | 1000 | -| | | | | | -| **all fields (micro avg.)** | **98.05** | **97.72** | **97.89** | 2938 | -| all fields (macro avg.) | 98.07 | 97.75 | 97.91 | 2938 | - - +| label | precision | recall | f1 | support | +|-----------------------------|-----------|-----------|-----------|---------| +| authors | 99.07 | 99.07 | 99.07 | 969 | +| first_author | 99.38 | 99.38 | 99.38 | 969 | +| title | 95.76 | 94.8 | 95.28 | 1000 | +| | | | | | +| **all fields (micro avg.)** | **98.05** | **97.72** | **97.89** | 2938 | +| all fields (macro avg.) | 98.07 | 97.75 | 97.91 | 2938 | #### Soft Matching (ignoring punctuation, case and space characters mismatches) **Field-level results** -| label | precision | recall | f1 | support | -|--- |--- |--- |--- |--- | -| authors | 99.07 | 99.07 | 99.07 | 969 | -| first_author | 99.38 | 99.38 | 99.38 | 969 | -| title | 99.29 | 98.3 | 98.79 | 1000 | -| | | | | | -| **all fields (micro avg.)** | **99.25** | **98.91** | **99.08** | 2938 | -| all fields (macro avg.) | 99.25 | 98.92 | 99.08 | 2938 | - - +| label | precision | recall | f1 | support | +|-----------------------------|-----------|-----------|-----------|---------| +| authors | 99.07 | 99.07 | 99.07 | 969 | +| first_author | 99.38 | 99.38 | 99.38 | 969 | +| title | 99.29 | 98.3 | 98.79 | 1000 | +| | | | | | +| **all fields (micro avg.)** | **99.25** | **98.91** | **99.08** | 2938 | +| all fields (macro avg.) | 99.25 | 98.92 | 99.08 | 2938 | #### Levenshtein Matching (Minimum Levenshtein distance at 0.8) **Field-level results** -| label | precision | recall | f1 | support | -|--- |--- |--- |--- |--- | -| authors | 99.38 | 99.38 | 99.38 | 969 | -| first_author | 99.48 | 99.48 | 99.48 | 969 | -| title | 99.7 | 98.7 | 99.2 | 1000 | -| | | | | | -| **all fields (micro avg.)** | **99.52** | **99.18** | **99.35** | 2938 | -| all fields (macro avg.) | 99.52 | 99.19 | 99.35 | 2938 | - - +| label | precision | recall | f1 | support | +|-----------------------------|-----------|-----------|-----------|---------| +| authors | 99.38 | 99.38 | 99.38 | 969 | +| first_author | 99.48 | 99.48 | 99.48 | 969 | +| title | 99.7 | 98.7 | 99.2 | 1000 | +| | | | | | +| **all fields (micro avg.)** | **99.52** | **99.18** | **99.35** | 2938 | +| all fields (macro avg.) | 99.52 | 99.19 | 99.35 | 2938 | #### Ratcliff/Obershelp Matching (Minimum Ratcliff/Obershelp similarity at 0.95) **Field-level results** -| label | precision | recall | f1 | support | -|--- |--- |--- |--- |--- | -| authors | 99.28 | 99.28 | 99.28 | 969 | -| first_author | 99.38 | 99.38 | 99.38 | 969 | -| title | 99.49 | 98.5 | 98.99 | 1000 | -| | | | | | -| **all fields (micro avg.)** | **99.39** | **99.05** | **99.22** | 2938 | -| all fields (macro avg.) | 99.38 | 99.05 | 99.22 | 2938 | - +| label | precision | recall | f1 | support | +|-----------------------------|-----------|-----------|-----------|---------| +| authors | 99.28 | 99.28 | 99.28 | 969 | +| first_author | 99.38 | 99.38 | 99.38 | 969 | +| title | 99.49 | 98.5 | 98.99 | 1000 | +| | | | | | +| **all fields (micro avg.)** | **99.39** | **99.05** | **99.22** | 2938 | +| all fields (macro avg.) | 99.38 | 99.05 | 99.22 | 2938 | #### Instance-level results @@ -77,8 +70,7 @@ Instance-level recall: 98.2 (Levenshtein) Instance-level recall: 98.1 (RatcliffObershelp) ``` - -## Citation metadata +## Citation metadata Evaluation on 1000 random PDF files out of 998 PDF (ratio 1.0). @@ -86,80 +78,73 @@ Evaluation on 1000 random PDF files out of 998 PDF (ratio 1.0). **Field-level results** -| label | precision | recall | f1 | support | -|--- |--- |--- |--- |--- | -| authors | 81.15 | 78.34 | 79.72 | 44770 | -| date | 84.59 | 81.14 | 82.83 | 45457 | -| first_author | 91.45 | 88.25 | 89.82 | 44770 | -| inTitle | 81.65 | 83.48 | 82.56 | 42795 | -| issue | 93.57 | 92.54 | 93.05 | 18983 | -| page | 93.68 | 77.5 | 84.82 | 40844 | -| title | 59.96 | 60.41 | 60.18 | 43101 | -| volume | 95.86 | 96.01 | 95.94 | 40458 | -| | | | | | -| **all fields (micro avg.)** | **84.21** | **81.35** | **82.76** | 321178 | -| all fields (macro avg.) | 85.24 | 82.21 | 83.62 | 321178 | - - +| label | precision | recall | f1 | support | +|-----------------------------|-----------|-----------|-----------|---------| +| authors | 81.15 | 78.34 | 79.72 | 44770 | +| date | 84.59 | 81.14 | 82.83 | 45457 | +| first_author | 91.45 | 88.25 | 89.82 | 44770 | +| inTitle | 81.65 | 83.48 | 82.56 | 42795 | +| issue | 93.57 | 92.54 | 93.05 | 18983 | +| page | 93.68 | 77.5 | 84.82 | 40844 | +| title | 59.96 | 60.41 | 60.18 | 43101 | +| volume | 95.86 | 96.01 | 95.94 | 40458 | +| | | | | | +| **all fields (micro avg.)** | **84.21** | **81.35** | **82.76** | 321178 | +| all fields (macro avg.) | 85.24 | 82.21 | 83.62 | 321178 | #### Soft Matching (ignoring punctuation, case and space characters mismatches) **Field-level results** -| label | precision | recall | f1 | support | -|--- |--- |--- |--- |--- | -| authors | 81.47 | 78.64 | 80.03 | 44770 | -| date | 84.59 | 81.14 | 82.83 | 45457 | -| first_author | 91.67 | 88.46 | 90.04 | 44770 | -| inTitle | 85.48 | 87.39 | 86.42 | 42795 | -| issue | 93.57 | 92.54 | 93.05 | 18983 | -| page | 93.68 | 77.5 | 84.82 | 40844 | -| title | 91.95 | 92.65 | 92.3 | 43101 | -| volume | 95.86 | 96.01 | 95.94 | 40458 | -| | | | | | -| **all fields (micro avg.)** | **89.3** | **86.27** | **87.76** | 321178 | -| all fields (macro avg.) | 89.78 | 86.79 | 88.18 | 321178 | - - +| label | precision | recall | f1 | support | +|-----------------------------|-----------|-----------|-----------|---------| +| authors | 81.47 | 78.64 | 80.03 | 44770 | +| date | 84.59 | 81.14 | 82.83 | 45457 | +| first_author | 91.67 | 88.46 | 90.04 | 44770 | +| inTitle | 85.48 | 87.39 | 86.42 | 42795 | +| issue | 93.57 | 92.54 | 93.05 | 18983 | +| page | 93.68 | 77.5 | 84.82 | 40844 | +| title | 91.95 | 92.65 | 92.3 | 43101 | +| volume | 95.86 | 96.01 | 95.94 | 40458 | +| | | | | | +| **all fields (micro avg.)** | **89.3** | **86.27** | **87.76** | 321178 | +| all fields (macro avg.) | 89.78 | 86.79 | 88.18 | 321178 | #### Levenshtein Matching (Minimum Levenshtein distance at 0.8) **Field-level results** -| label | precision | recall | f1 | support | -|--- |--- |--- |--- |--- | -| authors | 90.62 | 87.48 | 89.02 | 44770 | -| date | 84.59 | 81.14 | 82.83 | 45457 | -| first_author | 92.21 | 88.98 | 90.56 | 44770 | -| inTitle | 86.41 | 88.35 | 87.37 | 42795 | -| issue | 93.57 | 92.54 | 93.05 | 18983 | -| page | 93.68 | 77.5 | 84.82 | 40844 | -| title | 94.54 | 95.26 | 94.9 | 43101 | -| volume | 95.86 | 96.01 | 95.94 | 40458 | -| | | | | | -| **all fields (micro avg.)** | **91.15** | **88.06** | **89.57** | 321178 | -| all fields (macro avg.) | 91.44 | 88.41 | 89.81 | 321178 | - - +| label | precision | recall | f1 | support | +|-----------------------------|-----------|-----------|-----------|---------| +| authors | 90.62 | 87.48 | 89.02 | 44770 | +| date | 84.59 | 81.14 | 82.83 | 45457 | +| first_author | 92.21 | 88.98 | 90.56 | 44770 | +| inTitle | 86.41 | 88.35 | 87.37 | 42795 | +| issue | 93.57 | 92.54 | 93.05 | 18983 | +| page | 93.68 | 77.5 | 84.82 | 40844 | +| title | 94.54 | 95.26 | 94.9 | 43101 | +| volume | 95.86 | 96.01 | 95.94 | 40458 | +| | | | | | +| **all fields (micro avg.)** | **91.15** | **88.06** | **89.57** | 321178 | +| all fields (macro avg.) | 91.44 | 88.41 | 89.81 | 321178 | #### Ratcliff/Obershelp Matching (Minimum Ratcliff/Obershelp similarity at 0.95) **Field-level results** -| label | precision | recall | f1 | support | -|--- |--- |--- |--- |--- | -| authors | 84.91 | 81.97 | 83.41 | 44770 | -| date | 84.59 | 81.14 | 82.83 | 45457 | -| first_author | 91.45 | 88.25 | 89.82 | 44770 | -| inTitle | 85.13 | 87.04 | 86.07 | 42795 | -| issue | 93.57 | 92.54 | 93.05 | 18983 | -| page | 93.68 | 77.5 | 84.82 | 40844 | -| title | 93.93 | 94.64 | 94.28 | 43101 | -| volume | 95.86 | 96.01 | 95.94 | 40458 | -| | | | | | -| **all fields (micro avg.)** | **89.98** | **86.93** | **88.43** | 321178 | -| all fields (macro avg.) | 90.39 | 87.38 | 88.78 | 321178 | - +| label | precision | recall | f1 | support | +|-----------------------------|-----------|-----------|-----------|---------| +| authors | 84.91 | 81.97 | 83.41 | 44770 | +| date | 84.59 | 81.14 | 82.83 | 45457 | +| first_author | 91.45 | 88.25 | 89.82 | 44770 | +| inTitle | 85.13 | 87.04 | 86.07 | 42795 | +| issue | 93.57 | 92.54 | 93.05 | 18983 | +| page | 93.68 | 77.5 | 84.82 | 40844 | +| title | 93.93 | 94.64 | 94.28 | 43101 | +| volume | 95.86 | 96.01 | 95.94 | 40458 | +| | | | | | +| **all fields (micro avg.)** | **89.98** | **86.93** | **88.43** | 321178 | +| all fields (macro avg.) | 90.39 | 87.38 | 88.78 | 321178 | #### Instance-level results @@ -197,8 +182,8 @@ Matching 4 : 1801 Total matches : 41651 ``` - #### Citation context resolution + ``` Total expected references: 48449 - 48.45 references per article diff --git a/doc/benchmarks/flavors/article_light_ref/benchmaking-pmc.md b/doc/benchmarks/flavors/article_light_ref/benchmaking-pmc.md index 7926a14f1f..b118a7208d 100644 --- a/doc/benchmarks/flavors/article_light_ref/benchmaking-pmc.md +++ b/doc/benchmarks/flavors/article_light_ref/benchmaking-pmc.md @@ -1,5 +1,5 @@ -## Header metadata +## Header metadata Evaluation on 1943 random PDF files out of 1941 PDF (ratio 1.0). @@ -7,60 +7,53 @@ Evaluation on 1943 random PDF files out of 1941 PDF (ratio 1.0). **Field-level results** -| label | precision | recall | f1 | support | -|--- |--- |--- |--- |--- | -| authors | 92.14 | 91.86 | 92 | 1941 | -| first_author | 96.33 | 96.03 | 96.18 | 1941 | -| title | 84.4 | 83.53 | 83.96 | 1943 | -| | | | | | -| **all fields (micro avg.)** | **90.97** | **90.47** | **90.72** | 5825 | -| all fields (macro avg.) | 90.96 | 90.47 | 90.72 | 5825 | - - +| label | precision | recall | f1 | support | +|-----------------------------|-----------|-----------|-----------|---------| +| authors | 92.14 | 91.86 | 92 | 1941 | +| first_author | 96.33 | 96.03 | 96.18 | 1941 | +| title | 84.4 | 83.53 | 83.96 | 1943 | +| | | | | | +| **all fields (micro avg.)** | **90.97** | **90.47** | **90.72** | 5825 | +| all fields (macro avg.) | 90.96 | 90.47 | 90.72 | 5825 | #### Soft Matching (ignoring punctuation, case and space characters mismatches) **Field-level results** -| label | precision | recall | f1 | support | -|--- |--- |--- |--- |--- | -| authors | 94.06 | 93.77 | 93.91 | 1941 | -| first_author | 96.69 | 96.39 | 96.54 | 1941 | -| title | 92.1 | 91.15 | 91.62 | 1943 | -| | | | | | -| **all fields (micro avg.)** | **94.29** | **93.77** | **94.03** | 5825 | -| all fields (macro avg.) | 94.28 | 93.77 | 94.02 | 5825 | - - +| label | precision | recall | f1 | support | +|-----------------------------|-----------|-----------|-----------|---------| +| authors | 94.06 | 93.77 | 93.91 | 1941 | +| first_author | 96.69 | 96.39 | 96.54 | 1941 | +| title | 92.1 | 91.15 | 91.62 | 1943 | +| | | | | | +| **all fields (micro avg.)** | **94.29** | **93.77** | **94.03** | 5825 | +| all fields (macro avg.) | 94.28 | 93.77 | 94.02 | 5825 | #### Levenshtein Matching (Minimum Levenshtein distance at 0.8) **Field-level results** -| label | precision | recall | f1 | support | -|--- |--- |--- |--- |--- | -| authors | 96.38 | 96.08 | 96.23 | 1941 | -| first_author | 97 | 96.7 | 96.85 | 1941 | -| title | 98.23 | 97.22 | 97.72 | 1943 | -| | | | | | -| **all fields (micro avg.)** | **97.2** | **96.67** | **96.94** | 5825 | -| all fields (macro avg.) | 97.21 | 96.67 | 96.94 | 5825 | - - +| label | precision | recall | f1 | support | +|-----------------------------|-----------|-----------|-----------|---------| +| authors | 96.38 | 96.08 | 96.23 | 1941 | +| first_author | 97 | 96.7 | 96.85 | 1941 | +| title | 98.23 | 97.22 | 97.72 | 1943 | +| | | | | | +| **all fields (micro avg.)** | **97.2** | **96.67** | **96.94** | 5825 | +| all fields (macro avg.) | 97.21 | 96.67 | 96.94 | 5825 | #### Ratcliff/Obershelp Matching (Minimum Ratcliff/Obershelp similarity at 0.95) **Field-level results** -| label | precision | recall | f1 | support | -|--- |--- |--- |--- |--- | -| authors | 95.25 | 94.95 | 95.1 | 1941 | -| first_author | 96.33 | 96.03 | 96.18 | 1941 | -| title | 96.26 | 95.27 | 95.76 | 1943 | -| | | | | | -| **all fields (micro avg.)** | **95.94** | **95.42** | **95.68** | 5825 | -| all fields (macro avg.) | 95.94 | 95.42 | 95.68 | 5825 | - +| label | precision | recall | f1 | support | +|-----------------------------|-----------|-----------|-----------|---------| +| authors | 95.25 | 94.95 | 95.1 | 1941 | +| first_author | 96.33 | 96.03 | 96.18 | 1941 | +| title | 96.26 | 95.27 | 95.76 | 1943 | +| | | | | | +| **all fields (micro avg.)** | **95.94** | **95.42** | **95.68** | 5825 | +| all fields (macro avg.) | 95.94 | 95.42 | 95.68 | 5825 | #### Instance-level results @@ -77,8 +70,7 @@ Instance-level recall: 93.67 (Levenshtein) Instance-level recall: 90.74 (RatcliffObershelp) ``` - -## Citation metadata +## Citation metadata Evaluation on 1943 random PDF files out of 1941 PDF (ratio 1.0). @@ -86,80 +78,73 @@ Evaluation on 1943 random PDF files out of 1941 PDF (ratio 1.0). **Field-level results** -| label | precision | recall | f1 | support | -|--- |--- |--- |--- |--- | -| authors | 82.92 | 75.64 | 79.11 | 85778 | -| date | 94.33 | 83.47 | 88.57 | 87067 | -| first_author | 89.64 | 81.75 | 85.51 | 85778 | -| inTitle | 72.88 | 71.09 | 71.98 | 81007 | -| issue | 89.96 | 87.44 | 88.68 | 16635 | -| page | 94.06 | 82.82 | 88.08 | 80501 | -| title | 79.47 | 74.58 | 76.95 | 80736 | -| volume | 95.71 | 88.98 | 92.22 | 80067 | -| | | | | | -| **all fields (micro avg.)** | **86.93** | **79.98** | **83.31** | 597569 | -| all fields (macro avg.) | 87.37 | 80.72 | 83.89 | 597569 | - - +| label | precision | recall | f1 | support | +|-----------------------------|-----------|-----------|-----------|---------| +| authors | 82.92 | 75.64 | 79.11 | 85778 | +| date | 94.33 | 83.47 | 88.57 | 87067 | +| first_author | 89.64 | 81.75 | 85.51 | 85778 | +| inTitle | 72.88 | 71.09 | 71.98 | 81007 | +| issue | 89.96 | 87.44 | 88.68 | 16635 | +| page | 94.06 | 82.82 | 88.08 | 80501 | +| title | 79.47 | 74.58 | 76.95 | 80736 | +| volume | 95.71 | 88.98 | 92.22 | 80067 | +| | | | | | +| **all fields (micro avg.)** | **86.93** | **79.98** | **83.31** | 597569 | +| all fields (macro avg.) | 87.37 | 80.72 | 83.89 | 597569 | #### Soft Matching (ignoring punctuation, case and space characters mismatches) **Field-level results** -| label | precision | recall | f1 | support | -|--- |--- |--- |--- |--- | -| authors | 83.39 | 76.08 | 79.56 | 85778 | -| date | 94.33 | 83.47 | 88.57 | 87067 | -| first_author | 89.81 | 81.91 | 85.68 | 85778 | -| inTitle | 84.61 | 82.54 | 83.56 | 81007 | -| issue | 89.96 | 87.44 | 88.68 | 16635 | -| page | 94.06 | 82.82 | 88.08 | 80501 | -| title | 91.23 | 85.62 | 88.34 | 80736 | -| volume | 95.71 | 88.98 | 92.22 | 80067 | -| | | | | | -| **all fields (micro avg.)** | **90.33** | **83.11** | **86.57** | 597569 | -| all fields (macro avg.) | 90.39 | 83.61 | 86.84 | 597569 | - - +| label | precision | recall | f1 | support | +|-----------------------------|-----------|-----------|-----------|---------| +| authors | 83.39 | 76.08 | 79.56 | 85778 | +| date | 94.33 | 83.47 | 88.57 | 87067 | +| first_author | 89.81 | 81.91 | 85.68 | 85778 | +| inTitle | 84.61 | 82.54 | 83.56 | 81007 | +| issue | 89.96 | 87.44 | 88.68 | 16635 | +| page | 94.06 | 82.82 | 88.08 | 80501 | +| title | 91.23 | 85.62 | 88.34 | 80736 | +| volume | 95.71 | 88.98 | 92.22 | 80067 | +| | | | | | +| **all fields (micro avg.)** | **90.33** | **83.11** | **86.57** | 597569 | +| all fields (macro avg.) | 90.39 | 83.61 | 86.84 | 597569 | #### Levenshtein Matching (Minimum Levenshtein distance at 0.8) **Field-level results** -| label | precision | recall | f1 | support | -|--- |--- |--- |--- |--- | -| authors | 89.04 | 81.23 | 84.96 | 85778 | -| date | 94.33 | 83.47 | 88.57 | 87067 | -| first_author | 90.01 | 82.08 | 85.86 | 85778 | -| inTitle | 85.86 | 83.75 | 84.79 | 81007 | -| issue | 89.96 | 87.44 | 88.68 | 16635 | -| page | 94.06 | 82.82 | 88.08 | 80501 | -| title | 93.55 | 87.8 | 90.58 | 80736 | -| volume | 95.71 | 88.98 | 92.22 | 80067 | -| | | | | | -| **all fields (micro avg.)** | **91.66** | **84.33** | **87.84** | 597569 | -| all fields (macro avg.) | 91.56 | 84.7 | 87.97 | 597569 | - - +| label | precision | recall | f1 | support | +|-----------------------------|-----------|-----------|-----------|---------| +| authors | 89.04 | 81.23 | 84.96 | 85778 | +| date | 94.33 | 83.47 | 88.57 | 87067 | +| first_author | 90.01 | 82.08 | 85.86 | 85778 | +| inTitle | 85.86 | 83.75 | 84.79 | 81007 | +| issue | 89.96 | 87.44 | 88.68 | 16635 | +| page | 94.06 | 82.82 | 88.08 | 80501 | +| title | 93.55 | 87.8 | 90.58 | 80736 | +| volume | 95.71 | 88.98 | 92.22 | 80067 | +| | | | | | +| **all fields (micro avg.)** | **91.66** | **84.33** | **87.84** | 597569 | +| all fields (macro avg.) | 91.56 | 84.7 | 87.97 | 597569 | #### Ratcliff/Obershelp Matching (Minimum Ratcliff/Obershelp similarity at 0.95) **Field-level results** -| label | precision | recall | f1 | support | -|--- |--- |--- |--- |--- | -| authors | 85.85 | 78.32 | 81.91 | 85778 | -| date | 94.33 | 83.47 | 88.57 | 87067 | -| first_author | 89.66 | 81.77 | 85.53 | 85778 | -| inTitle | 83.2 | 81.15 | 82.16 | 81007 | -| issue | 89.96 | 87.44 | 88.68 | 16635 | -| page | 94.06 | 82.82 | 88.08 | 80501 | -| title | 93.15 | 87.42 | 90.2 | 80736 | -| volume | 95.71 | 88.98 | 92.22 | 80067 | -| | | | | | -| **all fields (micro avg.)** | **90.72** | **83.47** | **86.94** | 597569 | -| all fields (macro avg.) | 90.74 | 83.92 | 87.17 | 597569 | - +| label | precision | recall | f1 | support | +|-----------------------------|-----------|-----------|-----------|---------| +| authors | 85.85 | 78.32 | 81.91 | 85778 | +| date | 94.33 | 83.47 | 88.57 | 87067 | +| first_author | 89.66 | 81.77 | 85.53 | 85778 | +| inTitle | 83.2 | 81.15 | 82.16 | 81007 | +| issue | 89.96 | 87.44 | 88.68 | 16635 | +| page | 94.06 | 82.82 | 88.08 | 80501 | +| title | 93.15 | 87.42 | 90.2 | 80736 | +| volume | 95.71 | 88.98 | 92.22 | 80067 | +| | | | | | +| **all fields (micro avg.)** | **90.72** | **83.47** | **86.94** | 597569 | +| all fields (macro avg.) | 90.74 | 83.92 | 87.17 | 597569 | #### Instance-level results @@ -197,8 +182,8 @@ Matching 4 : 668 Total matches : 74329 ``` - #### Citation context resolution + ``` Total expected references: 90125 - 46.38 references per article