Skip to content

Commit b1cb6d3

Browse files
committed
Merge branch 'origin/master' into release/graal-vm/1.0
2 parents 0590b8b + d06ed1d commit b1cb6d3

File tree

1,254 files changed

+62357
-27021
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

1,254 files changed

+62357
-27021
lines changed

.gitignore

+1
Original file line numberDiff line numberDiff line change
@@ -50,6 +50,7 @@ language
5050
Python3.g4.stamp
5151
*.orig
5252
/graalpython/com.oracle.graal.python.test/src/tests/cpyext/Py*.c
53+
/graalpython/com.oracle.graal.python.test/src/tests/cpyext/Test*.c
5354
/*.diff
5455
## generated from: pyhocon -i ci.hocon -f json -o ci.json
5556
/ci.json

CHANGELOG.md

+11
Original file line numberDiff line numberDiff line change
@@ -3,6 +3,17 @@
33
This changelog summarizes major changes between GraalVM versions of the Python
44
language runtime. The main focus is on user-observable behavior of the engine.
55

6+
## Version 1.0.0 RC5
7+
8+
* Generator expressions now properly evaluate their first iterator in the definition scope at definition time
9+
* Fixes for embedders to ensure top scopes are stable and local scopes always contain TruffleObjects
10+
* C-API improvements to support simple Cython modules
11+
* Support recognition of Python source files with the polyglot launcher
12+
13+
## Version 1.0.0 RC4
14+
15+
* No changes
16+
617
## Version 1.0.0 RC3
718

819
* Support for more String encodings

CONTRIBUTING.md

+44
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,44 @@
1+
**Thank you for thinking about contributing something to this Python
2+
implementation on Graal!**
3+
4+
## First things first
5+
6+
You will need to sign the [Oracle Contributor
7+
Agreement](http://www.graalvm.org/community/contributors/) for us to be able to
8+
merge your work.
9+
10+
Please also take some time to review our [code of
11+
conduct](http://www.graalvm.org/community/conduct/) for contributors.
12+
13+
## How to contribute?
14+
15+
If you want to make a contribution, it's best to open an issue first about the
16+
area you would like to offer your help with. This implementation is in an early
17+
state and it will make more sense to work on some areas than others, so before
18+
you sink work into a pull request that we won't be able to accept, please talk
19+
to us :)
20+
21+
To get started with development on Python, first make sure you can build at
22+
least the _truffle_ project in the [Graal](https://github.com/oracle/graal)
23+
repository with our build tool [mx](https://github.com/graalvm/mx), as well as
24+
[Sulong](https://github.com/graalvm/sulong).
25+
26+
Once you can build those projects, clone this repository and run
27+
28+
$ mx build
29+
$ mx python -c "print(42)"
30+
31+
If this prints "42", then everything went fine and you just built and ran
32+
Python. Note that you don't need GraalVM for this, but then you'll only run
33+
interpreted, which won't give very good performance (but that might be fine for
34+
development).
35+
36+
When you make your changes, you can test them with `mx python`. Additionally,
37+
there are various "gates" that we use on our CI system to check any code that
38+
goes in. You can run all of these gates using `mx python-gate` or just some by
39+
using `mx python-gate --tags [TAG]`. Three interesting tags to run that cover
40+
most things are:
41+
42+
- `python-unittest` - Run the unittests written in Python, including those for the C extension API
43+
- `python-graalvm` - Build a minimal GraalVM bundle with Python only and a native launcher
44+
- `python-license` - Check that all files have the correct copyright headers applied to them

README.md

+1-1
Original file line numberDiff line numberDiff line change
@@ -22,4 +22,4 @@ Universal Permissive License v 1.0 as shown at
2222
[http://oss.oracle.com/licenses/upl](http://oss.oracle.com/licenses/upl). This
2323
implementation is in part derived from and contains additional code from 3rd
2424
parties, the copyrights and licensing of which is detailed in the
25-
[LICENSE](LICENSE) file.
25+
[LICENSE](LICENSE) and [3rd_party_licenses.txt](3rd_party_licenses.txt) files.

ci.jsonnet

+10-4
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
{
2-
overlay: "03b824f947adf9a67c80c3e8a27646d43a687bfb",
2+
overlay: "2562524065c42e18d4a5ea57e8cbf6cac6b9bdda",
33

44
// ======================================================================================================
55
//
@@ -53,6 +53,9 @@
5353
error "unknown field: "+field+" in "+object+", valid choices are: "+std.objectFields(object)
5454
else
5555
object[field],
56+
57+
graalOption: function(name, value)
58+
["--Ja", "@-Dgraal."+name+"="+value],
5659
},
5760

5861
// ------------------------------------------------------------------------------------------------------
@@ -79,7 +82,7 @@
7982
linuxMixin: linuxMixin,
8083

8184
local linuxBenchMixin = linuxMixin + {
82-
capabilities +: ["no_frequency_scaling", "tmpfs25g", "x62"],
85+
capabilities +: ["no_frequency_scaling", "tmpfs25g", "x52"],
8386
},
8487
linuxBenchMixin: linuxBenchMixin,
8588

@@ -178,7 +181,7 @@
178181

179182
setup +: [
180183
["mx", "sforceimport"],
181-
["mx", "-v", "--dynamicimports", self.dynamicImports, "build"],
184+
["mx", "--dynamicimports", self.dynamicImports, "build"],
182185
]
183186
},
184187
commonBuilder: commonBuilder,
@@ -191,9 +194,12 @@
191194
local baseGate = commonBuilder + {
192195
tags: "tags must be defined",
193196

197+
// local truffleDebugFlags = utils.graalOption("TraceTruffleCompilation", "true"),
198+
// local truffleDebugFlags = utils.graalOption("TraceTruffleCompilationDetails", "true"),
199+
local truffleDebugFlags = [],
194200
targets: TARGET.gate,
195201
run +: [
196-
["mx", "--strict-compliance", "--dynamicimports", super.dynamicImports, "--primary", "gate", "--tags", self.tags, "-B=--force-deprecation-as-warning-for-dependencies"],
202+
["mx"] + truffleDebugFlags + ["--strict-compliance", "--dynamicimports", super.dynamicImports, "--primary", "gate", "--tags", self.tags, "-B=--force-deprecation-as-warning-for-dependencies"],
197203
]
198204
},
199205

graalpython/benchmarks/src/benchmarks/interop/python_r_image_demo.py

+11-9
Original file line numberDiff line numberDiff line change
@@ -1,19 +1,21 @@
1-
# Copyright (c) 2018, Oracle and/or its affiliates.
1+
# Copyright (c) 2018, Oracle and/or its affiliates. All rights reserved.
2+
# DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
23
#
34
# The Universal Permissive License (UPL), Version 1.0
45
#
56
# Subject to the condition set forth below, permission is hereby granted to any
6-
# person obtaining a copy of this software, associated documentation and/or data
7-
# (collectively the "Software"), free of charge and under any and all copyright
8-
# rights in the Software, and any and all patent rights owned or freely
9-
# licensable by each licensor hereunder covering either (i) the unmodified
10-
# Software as contributed to or provided by such licensor, or (ii) the Larger
11-
# Works (as defined below), to deal in both
7+
# person obtaining a copy of this software, associated documentation and/or
8+
# data (collectively the "Software"), free of charge and under any and all
9+
# copyright rights in the Software, and any and all patent rights owned or
10+
# freely licensable by each licensor hereunder covering either (i) the
11+
# unmodified Software as contributed to or provided by such licensor, or (ii)
12+
# the Larger Works (as defined below), to deal in both
1213
#
1314
# (a) the Software, and
15+
#
1416
# (b) any piece of software and/or hardware listed in the lrgrwrks.txt file if
15-
# one is included with the Software (each a "Larger Work" to which the
16-
# Software is contributed by such licensors),
17+
# one is included with the Software each a "Larger Work" to which the Software
18+
# is contributed by such licensors),
1719
#
1820
# without restriction, including without limitation the rights to copy, create
1921
# derivative works of, display, perform, and distribute the Software and make,

graalpython/benchmarks/src/benchmarks/interop/r_python_image_demo.r

+11-9
Original file line numberDiff line numberDiff line change
@@ -1,19 +1,21 @@
1-
# Copyright (c) 2018, Oracle and/or its affiliates.
1+
# Copyright (c) 2018, Oracle and/or its affiliates. All rights reserved.
2+
# DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
23
#
34
# The Universal Permissive License (UPL), Version 1.0
45
#
56
# Subject to the condition set forth below, permission is hereby granted to any
6-
# person obtaining a copy of this software, associated documentation and/or data
7-
# (collectively the "Software"), free of charge and under any and all copyright
8-
# rights in the Software, and any and all patent rights owned or freely
9-
# licensable by each licensor hereunder covering either (i) the unmodified
10-
# Software as contributed to or provided by such licensor, or (ii) the Larger
11-
# Works (as defined below), to deal in both
7+
# person obtaining a copy of this software, associated documentation and/or
8+
# data (collectively the "Software"), free of charge and under any and all
9+
# copyright rights in the Software, and any and all patent rights owned or
10+
# freely licensable by each licensor hereunder covering either (i) the
11+
# unmodified Software as contributed to or provided by such licensor, or (ii)
12+
# the Larger Works (as defined below), to deal in both
1213
#
1314
# (a) the Software, and
15+
#
1416
# (b) any piece of software and/or hardware listed in the lrgrwrks.txt file if
15-
# one is included with the Software (each a "Larger Work" to which the
16-
# Software is contributed by such licensors),
17+
# one is included with the Software each a "Larger Work" to which the Software
18+
# is contributed by such licensors),
1719
#
1820
# without restriction, including without limitation the rights to copy, create
1921
# derivative works of, display, perform, and distribute the Software and make,

graalpython/benchmarks/src/benchmarks/sieve.py

+11-9
Original file line numberDiff line numberDiff line change
@@ -1,19 +1,21 @@
1-
# Copyright (c) 2017, 2018, Oracle and/or its affiliates.
1+
# Copyright (c) 2017, 2018, Oracle and/or its affiliates. All rights reserved.
2+
# DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
23
#
34
# The Universal Permissive License (UPL), Version 1.0
45
#
56
# Subject to the condition set forth below, permission is hereby granted to any
6-
# person obtaining a copy of this software, associated documentation and/or data
7-
# (collectively the "Software"), free of charge and under any and all copyright
8-
# rights in the Software, and any and all patent rights owned or freely
9-
# licensable by each licensor hereunder covering either (i) the unmodified
10-
# Software as contributed to or provided by such licensor, or (ii) the Larger
11-
# Works (as defined below), to deal in both
7+
# person obtaining a copy of this software, associated documentation and/or
8+
# data (collectively the "Software"), free of charge and under any and all
9+
# copyright rights in the Software, and any and all patent rights owned or
10+
# freely licensable by each licensor hereunder covering either (i) the
11+
# unmodified Software as contributed to or provided by such licensor, or (ii)
12+
# the Larger Works (as defined below), to deal in both
1213
#
1314
# (a) the Software, and
15+
#
1416
# (b) any piece of software and/or hardware listed in the lrgrwrks.txt file if
15-
# one is included with the Software (each a "Larger Work" to which the
16-
# Software is contributed by such licensors),
17+
# one is included with the Software each a "Larger Work" to which the Software
18+
# is contributed by such licensors),
1719
#
1820
# without restriction, including without limitation the rights to copy, create
1921
# derivative works of, display, perform, and distribute the Software and make,

graalpython/benchmarks/src/harness.py

+159
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,159 @@
1+
# Copyright (c) 2018, Oracle and/or its affiliates. All rights reserved.
2+
# DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
3+
#
4+
# The Universal Permissive License (UPL), Version 1.0
5+
#
6+
# Subject to the condition set forth below, permission is hereby granted to any
7+
# person obtaining a copy of this software, associated documentation and/or
8+
# data (collectively the "Software"), free of charge and under any and all
9+
# copyright rights in the Software, and any and all patent rights owned or
10+
# freely licensable by each licensor hereunder covering either (i) the
11+
# unmodified Software as contributed to or provided by such licensor, or (ii)
12+
# the Larger Works (as defined below), to deal in both
13+
#
14+
# (a) the Software, and
15+
#
16+
# (b) any piece of software and/or hardware listed in the lrgrwrks.txt file if
17+
# one is included with the Software each a "Larger Work" to which the Software
18+
# is contributed by such licensors),
19+
#
20+
# without restriction, including without limitation the rights to copy, create
21+
# derivative works of, display, perform, and distribute the Software and make,
22+
# use, sell, offer for sale, import, export, have made, and have sold the
23+
# Software and the Larger Work(s), and to sublicense the foregoing rights on
24+
# either these or other terms.
25+
#
26+
# This license is subject to the following condition:
27+
#
28+
# The above copyright notice and either this complete permission notice or at a
29+
# minimum a reference to the UPL must be included in all copies or substantial
30+
# portions of the Software.
31+
#
32+
# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
33+
# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
34+
# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
35+
# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
36+
# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
37+
# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
38+
# SOFTWARE.
39+
40+
import _io
41+
import os
42+
import sys
43+
from time import time
44+
45+
_HRULE = '-'.join(['' for i in range(80)])
46+
ATTR_BENCHMARK = '__benchmark__'
47+
ATTR_PROCESS_ARGS = '__process_args__'
48+
49+
50+
def _as_int(value):
51+
if isinstance(value, (list, tuple)):
52+
value = value[0]
53+
54+
if not isinstance(value, int):
55+
return int(value)
56+
return value
57+
58+
59+
class BenchRunner(object):
60+
def __init__(self, bench_file, bench_args=None, iterations=1, warmup=0):
61+
if bench_args is None:
62+
bench_args = []
63+
self.bench_module = BenchRunner.get_bench_module(bench_file)
64+
self.bench_args = bench_args
65+
self.iterations = _as_int(iterations)
66+
assert isinstance(self.iterations, int)
67+
self.warmup = _as_int(warmup)
68+
assert isinstance(self.warmup, int)
69+
70+
@staticmethod
71+
def get_bench_module(bench_file):
72+
name = bench_file.rpartition("/")[2].partition(".")[0].replace('.py', '')
73+
directory = bench_file.rpartition("/")[0]
74+
pkg = []
75+
while any(f.endswith("__init__.py") for f in os.listdir(directory)):
76+
directory, slash, postfix = directory.rpartition("/")
77+
pkg.insert(0, postfix)
78+
79+
if pkg:
80+
sys.path.insert(0, directory)
81+
bench_module = __import__(".".join(pkg + [name]))
82+
for p in pkg[1:]:
83+
bench_module = getattr(bench_module, p)
84+
bench_module = getattr(bench_module, name)
85+
return bench_module
86+
87+
else:
88+
bench_module = type(sys)(name, bench_file)
89+
with _io.FileIO(bench_file, "r") as f:
90+
bench_module.__file__ = bench_file
91+
exec(compile(f.readall(), bench_file, "exec"), bench_module.__dict__)
92+
return bench_module
93+
94+
def _get_attr(self, attr_name):
95+
if hasattr(self.bench_module, attr_name):
96+
return getattr(self.bench_module, attr_name)
97+
98+
def _call_attr(self, attr_name, *args):
99+
attr = self._get_attr(attr_name)
100+
if attr and hasattr(attr, '__call__'):
101+
return attr(*args)
102+
103+
def run(self):
104+
print(_HRULE)
105+
print("### %s, %s warmup iterations, %s bench iterations " % (self.bench_module.__name__, self.warmup, self.iterations))
106+
107+
# process the args if the processor function is defined
108+
args = self._call_attr(ATTR_PROCESS_ARGS, *self.bench_args)
109+
if args is None:
110+
# default args processor considers all args as ints
111+
args = list(map(int, self.bench_args))
112+
113+
print("### args = %s" % args)
114+
print(_HRULE)
115+
116+
bench_func = self._get_attr(ATTR_BENCHMARK)
117+
if bench_func and hasattr(bench_func, '__call__'):
118+
if self.warmup:
119+
print("### warming up for %s iterations ... " % self.warmup)
120+
for _ in range(self.warmup):
121+
bench_func(*args)
122+
123+
for iteration in range(self.iterations):
124+
start = time()
125+
bench_func(*args)
126+
duration = "%.3f" % (time() - start)
127+
print("### iteration=%s, name=%s, duration=%s" % (iteration, self.bench_module.__name__, duration))
128+
129+
130+
def run_benchmark(prog, args):
131+
warmup = 0
132+
iterations = 1
133+
bench_file = None
134+
bench_args = []
135+
136+
i = 0
137+
while i < len(args):
138+
arg = args[i]
139+
if arg == '-i':
140+
i += 1
141+
iterations = _as_int(args[i])
142+
elif arg.startswith("--iterations"):
143+
iterations = _as_int(arg.split("=")[1])
144+
elif arg == '-w':
145+
i += 1
146+
warmup = _as_int(args[i])
147+
elif arg.startswith("--warmup"):
148+
warmup = _as_int(arg.split("=")[1])
149+
elif bench_file is None:
150+
bench_file = arg
151+
else:
152+
bench_args.append(arg)
153+
i += 1
154+
155+
BenchRunner(bench_file, bench_args=bench_args, iterations=iterations, warmup=warmup).run()
156+
157+
158+
if __name__ == '__main__':
159+
run_benchmark(sys.argv[0], sys.argv[1:])

0 commit comments

Comments
 (0)