Skip to content

Update insertion benchmark #144

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 4 commits into from
May 28, 2024

Conversation

gmile
Copy link
Contributor

@gmile gmile commented May 3, 2024

This refreshes insertion benchmark stats + does a couple of minor fixes to the bench README.md.

I couldn't update all benchmarks due to a failure that I haven't had time to look into. Here's a full failure output:

Operating System: macOS
CPU Information: Apple M3 Max
Number of Available Cores: 16
Available memory: 128 GB
Elixir 1.16.2
Erlang 26.2.4
JIT enabled: true

Benchmark suite executing with the following configuration:
warmup: 2 s
time: 5 s
memory time: 0 ns
reduction time: 0 ns
parallel: 1
inputs: Big 1 Million, Date attr, Medium 100 Thousand, Small 1 Thousand, Time attr, UUID attr
Estimated total run time: 2 min 6 s

Benchmarking MyXQL Loader with input Big 1 Million ...
Benchmarking MyXQL Loader with input Date attr ...
Benchmarking MyXQL Loader with input Medium 100 Thousand ...
Benchmarking MyXQL Loader with input Small 1 Thousand ...
Benchmarking MyXQL Loader with input Time attr ...
Benchmarking MyXQL Loader with input UUID attr ...
Benchmarking Pg Loader with input Big 1 Million ...
Benchmarking Pg Loader with input Date attr ...
Benchmarking Pg Loader with input Medium 100 Thousand ...
Benchmarking Pg Loader with input Small 1 Thousand ...
Benchmarking Pg Loader with input Time attr ...
Benchmarking Pg Loader with input UUID attr ...
Benchmarking SQLite3 Loader with input Big 1 Million ...
Benchmarking SQLite3 Loader with input Date attr ...
Benchmarking SQLite3 Loader with input Medium 100 Thousand ...
Benchmarking SQLite3 Loader with input Small 1 Thousand ...
Benchmarking SQLite3 Loader with input Time attr ...

09:27:11.506 [error] Task #PID<0.352.0> started from #PID<0.98.0> terminating
** (FunctionClauseError) no function clause matching in Calendar.ISO.parse_time/1
    (elixir 1.16.2) lib/calendar/iso.ex:297: Calendar.ISO.parse_time(~T[21:25:04.361140])
    (elixir 1.16.2) lib/calendar/time.ex:255: Time.from_iso8601/2
    (ecto_sqlite3 0.15.1) lib/ecto/adapters/sqlite3/codec.ex:83: Ecto.Adapters.SQLite3.Codec.time_decode/1
    (ecto 3.11.1) lib/ecto/type.ex:932: Ecto.Type.process_loaders/3
    (ecto 3.11.1) lib/ecto/schema/loader.ex:71: anonymous fn/5 in Ecto.Schema.Loader.unsafe_load/4
    (elixir 1.16.2) lib/enum.ex:2528: Enum."-reduce/3-lists^foldl/2-0-"/3
    (elixir 1.16.2) lib/enum.ex:1700: Enum."-map/2-lists^map/1-1-"/2
    (benchee 1.3.0) lib/benchee/benchmark/collect/time.ex:17: Benchee.Benchmark.Collect.Time.collect/1
Function: #Function<2.121299024/0 in Benchee.Utility.Parallel.map/2>
    Args: []
** (EXIT from #PID<0.98.0>) an exception was raised:
    ** (FunctionClauseError) no function clause matching in Calendar.ISO.parse_time/1
        (elixir 1.16.2) lib/calendar/iso.ex:297: Calendar.ISO.parse_time(~T[21:25:04.361140])
        (elixir 1.16.2) lib/calendar/time.ex:255: Time.from_iso8601/2
        (ecto_sqlite3 0.15.1) lib/ecto/adapters/sqlite3/codec.ex:83: Ecto.Adapters.SQLite3.Codec.time_decode/1
        (ecto 3.11.1) lib/ecto/type.ex:932: Ecto.Type.process_loaders/3
        (ecto 3.11.1) lib/ecto/schema/loader.ex:71: anonymous fn/5 in Ecto.Schema.Loader.unsafe_load/4
        (elixir 1.16.2) lib/enum.ex:2528: Enum."-reduce/3-lists^foldl/2-0-"/3
        (elixir 1.16.2) lib/enum.ex:1700: Enum."-map/2-lists^map/1-1-"/2
        (benchee 1.3.0) lib/benchee/benchmark/collect/time.ex:17: Benchee.Benchmark.Collect.Time.collect/1

gmile added 2 commits May 3, 2024 09:37
MIX_ENV=bench doesn't exist, and postgrex and myxql are installed for
MIX_ENV=dev, so we can simplify this
5.7 won't start on arm64 architecture
@warmwaffles
Copy link
Member

@gmile you've stumbled on a bug.

How did you run the benchmarks. Were you running MySQL and Postgres locally vs running them in docker?

Overall thanks for bumping the benchmarks!

@gmile
Copy link
Contributor Author

gmile commented May 4, 2024

@warmwaffles to your question:

How did you run the benchmarks. Were you running MySQL and Postgres locally vs running them in docker?

I ran MySQL in a container, but PostgreSQL locally (natively, running the the host OS).

Your question made me realise its not fair to compare results of SQLite3 and PostgreSQL running on my computer natively with results for MySQL running inside a container, e.g. via virtualization which has a performance penalty. I'm using orbstack which claims to be faster the Docker, but still. I'll install MySQL on my mac natively and will regenerate the benchmark once again.

On a separately note, to clarify existence of commit a9902f3. Using docker image for MySQL 5.7 didn't work for me, because there's no arm64 OCI manifest generated for 5.7 version, but there's for 8.

@gmile
Copy link
Contributor Author

gmile commented May 4, 2024

I was able to reproduce the exception mentioned in my message with this self-contained Elixir script:

Mix.install([:ecto_sqlite3])

Application.put_env(:foo, Repo, pool_size: 1, database: ":memory:")

defmodule Repo do
  use Ecto.Repo,
    adapter: Ecto.Adapters.SQLite3,
    otp_app: :foo
end

defmodule Migration0 do
  use Ecto.Migration

  def change do
    create table("posts") do
      add(:title, :string)
      add(:published_at, :time)
    end
  end
end

defmodule Post do
  use Ecto.Schema

  schema "posts" do
    field(:title, :string)
    field(:published_at, :time)
  end
end

defmodule Main do
  import Ecto.Query, warn: false

  def reproduce do
    :ok = Repo.__adapter__().storage_up(Repo.config())

    {:ok, _pid} = Supervisor.start_link([Repo], strategy: :one_for_one)

    Ecto.Migrator.run(Repo, [{0, Migration0}], :up, all: true, log_migrations_sql: :debug)

    Repo.load(Post, %{title: "New blog post", published_at: ~T[21:25:04.361140]})
  end
end

Main.reproduce()

If I make a change to a script - it will no longer raise:

diff --git a/time-issue-repro.exs b/time-issue-repro.exs
index 8643d65..52e173e 100644
--- a/time-issue-repro.exs
+++ b/time-issue-repro.exs
@@ -38,7 +38,7 @@ defmodule Main do

     Ecto.Migrator.run(Repo, [{0, Migration0}], :up, all: true, log_migrations_sql: :debug)

-    Repo.load(Post, %{title: "New blog post", published_at: ~T[21:25:04.361140]})
+    Repo.load(Post, %{title: "New blog post", published_at: "21:25:04.361140")
   end
 end

@gmile gmile force-pushed the refresh-insert-benchmark branch from 43a2ee6 to 117488c Compare May 4, 2024 10:52
@gmile
Copy link
Contributor Author

gmile commented May 4, 2024

@warmwaffles I added this commit bceff9a just to see if I am able to get pass the issue when running the benchmark

It work, so not only "insert" part, but all benchmarks were regenerated successfully. However, I don't believe my fix is correct. For example, running benchmark having %Date{} struct as a value appears to run just fine without any changes:

"Date attr" =>
1..100_000 |> Enum.map(fn _ -> %{name: "Alice", date_attr: ~D[2018-06-20]} end),

I am not sure why explicit handling of %Time{} is necessary in principle 🤔

@warmwaffles warmwaffles merged commit 802fbf6 into elixir-sqlite:main May 28, 2024
10 checks passed
@warmwaffles
Copy link
Member

Sorry this took so long to merge. It completely slipped through on me. Feel free to ping mention me if you see another PR for these libraries fall through.

@gmile gmile deleted the refresh-insert-benchmark branch May 28, 2024 09:23
@gmile
Copy link
Contributor Author

gmile commented May 28, 2024

@warmwaffles noted, and no worries, I kind of forgot about this one too 🙈 Thanks for merging! I just hope this change is not going to break anything bceff9a 🙏

@warmwaffles
Copy link
Member

It shouldn't. You added a test for it and I think it was just an oversight by me initially.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants