Skip to content

Commit 0f7b746

Browse files
abhik1505040abhik1505040
committed
Added files
0 parents  commit 0f7b746

File tree

8 files changed

+1063
-0
lines changed

8 files changed

+1063
-0
lines changed

‎.gitignore

+164
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,164 @@
1+
# Initially taken from Github's Python gitignore file
2+
3+
# Byte-compiled / optimized / DLL files
4+
__pycache__/
5+
*.py[cod]
6+
*$py.class
7+
8+
# C extensions
9+
*.so
10+
11+
# tests and logs
12+
tests/fixtures/*
13+
!tests/fixtures/sample_text_no_unicode.txt
14+
logs/
15+
lightning_logs/
16+
lang_code_data/
17+
18+
# Distribution / packaging
19+
.Python
20+
build/
21+
develop-eggs/
22+
dist/
23+
downloads/
24+
eggs/
25+
.eggs/
26+
lib/
27+
lib64/
28+
parts/
29+
sdist/
30+
var/
31+
wheels/
32+
*.egg-info/
33+
.installed.cfg
34+
*.egg
35+
MANIFEST
36+
37+
# PyInstaller
38+
# Usually these files are written by a python script from a template
39+
# before PyInstaller builds the exe, so as to inject date/other infos into it.
40+
*.manifest
41+
*.spec
42+
43+
# Installer logs
44+
pip-log.txt
45+
pip-delete-this-directory.txt
46+
47+
# Unit test / coverage reports
48+
htmlcov/
49+
.tox/
50+
.nox/
51+
.coverage
52+
.coverage.*
53+
.cache
54+
nosetests.xml
55+
coverage.xml
56+
*.cover
57+
.hypothesis/
58+
.pytest_cache/
59+
60+
# Translations
61+
*.mo
62+
*.pot
63+
64+
# Django stuff:
65+
*.log
66+
local_settings.py
67+
db.sqlite3
68+
69+
# Flask stuff:
70+
instance/
71+
.webassets-cache
72+
73+
# Scrapy stuff:
74+
.scrapy
75+
76+
# Sphinx documentation
77+
docs/_build/
78+
79+
# PyBuilder
80+
target/
81+
82+
# Jupyter Notebook
83+
.ipynb_checkpoints
84+
85+
# IPython
86+
profile_default/
87+
ipython_config.py
88+
89+
# pyenv
90+
.python-version
91+
92+
# celery beat schedule file
93+
celerybeat-schedule
94+
95+
# SageMath parsed files
96+
*.sage.py
97+
98+
# Environments
99+
.env
100+
.venv
101+
env/
102+
venv/
103+
ENV/
104+
env.bak/
105+
venv.bak/
106+
107+
# Spyder project settings
108+
.spyderproject
109+
.spyproject
110+
111+
# Rope project settings
112+
.ropeproject
113+
114+
# mkdocs documentation
115+
/site
116+
117+
# mypy
118+
.mypy_cache/
119+
.dmypy.json
120+
dmypy.json
121+
122+
# Pyre type checker
123+
.pyre/
124+
125+
# vscode
126+
.vs
127+
.vscode
128+
129+
# Pycharm
130+
.idea
131+
132+
# TF code
133+
tensorflow_code
134+
135+
# Models
136+
proc_data
137+
138+
# examples
139+
runs
140+
/runs_old
141+
/wandb
142+
/examples/runs
143+
/examples/**/*.args
144+
/examples/rag/sweep
145+
146+
# data
147+
/data
148+
serialization_dir
149+
150+
# emacs
151+
*.*~
152+
debug.env
153+
154+
# vim
155+
.*.swp
156+
157+
#ctags
158+
tags
159+
160+
# pre-commit
161+
.pre-commit*
162+
163+
# .lock
164+
*.lock

‎LICENSE

+22
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,22 @@
1+
2+
MIT License
3+
4+
Copyright (c) 2021 Abhik Bhattacharjee
5+
6+
Permission is hereby granted, free of charge, to any person obtaining a copy
7+
of this software and associated documentation files (the "Software"), to deal
8+
in the Software without restriction, including without limitation the rights
9+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
10+
copies of the Software, and to permit persons to whom the Software is
11+
furnished to do so, subject to the following conditions:
12+
13+
The above copyright notice and this permission notice shall be included in all
14+
copies or substantial portions of the Software.
15+
16+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
17+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
18+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
19+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
20+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
21+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
22+
SOFTWARE.

‎README.md

+48
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,48 @@
1+
# normalizer
2+
3+
This python module is an easy-to-use port of the text normalization used in the paper **"[Not low-resource anymore: Aligner ensembling, batch filtering, and new datasets for Bengali-English machine translation](https://aclanthology.org/2020.emnlp-main.207/)".** It is intended to be used for normalizing / cleaning Bengali and English text.
4+
5+
## Installation
6+
```bash
7+
$ pip install git+https://github.com/csebuetnlp/normalizer
8+
```
9+
10+
## Usage
11+
12+
```python
13+
from normalizer import normalize
14+
input_text = """your input text"""
15+
normalized_text = normalize(
16+
input_text,
17+
unicode_norm="NFKC", # type of unicode normalization (default "NFKC")
18+
punct_replacement=None, # an optional string or callable for replacing the punctuations (default `None`, i.e. no replacement)
19+
url_replacement=None, # an optional string or callable for replacing the URLS (default `None`, i.e. no replacement)
20+
emoji_replacement=None, # an optional string or callable for replacing the emojis (default `None`, i.e. no replacement)
21+
apply_unicode_norm_last=True # whether to apply the unicode normalization before or after rule based replacements (default True)
22+
)
23+
```
24+
25+
## Citation
26+
27+
If you use this module in your work, please cite the following paper:
28+
```
29+
@inproceedings{hasan-etal-2020-low,
30+
title = "Not Low-Resource Anymore: Aligner Ensembling, Batch Filtering, and New Datasets for {B}engali-{E}nglish Machine Translation",
31+
author = "Hasan, Tahmid and
32+
Bhattacharjee, Abhik and
33+
Samin, Kazi and
34+
Hasan, Masum and
35+
Basak, Madhusudan and
36+
Rahman, M. Sohel and
37+
Shahriyar, Rifat",
38+
booktitle = "Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)",
39+
month = nov,
40+
year = "2020",
41+
address = "Online",
42+
publisher = "Association for Computational Linguistics",
43+
url = "https://www.aclweb.org/anthology/2020.emnlp-main.207",
44+
doi = "10.18653/v1/2020.emnlp-main.207",
45+
pages = "2612--2623",
46+
abstract = "Despite being the seventh most widely spoken language in the world, Bengali has received much less attention in machine translation literature due to being low in resources. Most publicly available parallel corpora for Bengali are not large enough; and have rather poor quality, mostly because of incorrect sentence alignments resulting from erroneous sentence segmentation, and also because of a high volume of noise present in them. In this work, we build a customized sentence segmenter for Bengali and propose two novel methods for parallel corpus creation on low-resource setups: aligner ensembling and batch filtering. With the segmenter and the two methods combined, we compile a high-quality Bengali-English parallel corpus comprising of 2.75 million sentence pairs, more than 2 million of which were not available before. Training on neural models, we achieve an improvement of more than 9 BLEU score over previous approaches to Bengali-English machine translation. We also evaluate on a new test set of 1000 pairs made with extensive quality control. We release the segmenter, parallel corpus, and the evaluation set, thus elevating Bengali from its low-resource status. To the best of our knowledge, this is the first ever large scale study on Bengali-English machine translation. We believe our study will pave the way for future research on Bengali-English machine translation as well as other low-resource languages. Our data and code are available at https://github.com/csebuetnlp/banglanmt.",
47+
}
48+
```

‎normalizer/__init__.py

+3
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
__version__ = "0.0.1"
2+
3+
from .normalize import *

0 commit comments

Comments
 (0)