Limit and/or Cache API requests to avoid: 403 - API rate limit exceeded for installation ID ___

[trinitronx]

I just switched to using Jekyll 4.3.2 and tried to deploy a static site using this theme, which lists all current GitHub repositories on the generated static site page. To do this, it uses jekyll-github-metadata gem (version 2.16.0) to gather all repositories for my GitHub user.

Apparently, this results in a lot of API requests (for my user, over ~486!) and quickly hits the rate limit when using GITHUB_TOKEN in a GitHub Actions workflow.

Would it be possible to cache or limit API requests further so static site deployments will complete successfully in GitHub Actions?

For example: The jekyll build --verbose output shows each repo is queried for both contributors and releases, but the template doesn't even need this information

  Logging at level: debug
    Jekyll Version: 4.3.2
Configuration file: /mnt/terabyte/src/sites/trinitronx.github.io/_config.yml
Configuration file: /mnt/terabyte/src/sites/trinitronx.github.io/_config_dev.yml
         Requiring: jekyll-github-metadata
To use retry middleware with Faraday v2.0+, install `faraday-retry` gem
   GitHub Metadata: Initializing...
            Source: /mnt/terabyte/src/sites/trinitronx.github.io
       Destination: /mnt/terabyte/src/sites/trinitronx.github.io/_site
 Incremental build: enabled
      Generating... 
           Reading: /_layouts/default.html
       EntryFilter: excluded /LICENSE.neumorphism
       EntryFilter: excluded /README.md
       EntryFilter: excluded /docs
       EntryFilter: excluded /package.json
       EntryFilter: excluded /node_modules
       EntryFilter: excluded /.jekyll-cache
       EntryFilter: excluded /.github
       EntryFilter: excluded /.ruby-version
       EntryFilter: excluded /Gemfile
       EntryFilter: excluded /yarn.lock
       EntryFilter: excluded /.ruby-gemset
       EntryFilter: excluded /Gemfile.lock
       EntryFilter: excluded /.gitignore
       EntryFilter: excluded /gulpfile.js
           Reading: index.html
           Reading: _data/projects.yml
           Reading: _data/skills-frameworks.yml
           Reading: _data/skills-languages.yml
           Reading: _data/skills-tools.yml
           Reading: _data/timeline.yml
         Rendering: index.html
  Pre-Render Hooks: index.html
  Rendering Markup: index.html
Post-Convert Hooks: index.html
  Rendering Layout: index.html
   GitHub Metadata: Generating for trinitronx/trinitronx.github.io
   GitHub Metadata: Calling @client.list_repos("trinitronx", {:type=>"public", :accept=>"application/vnd.github.mercy-preview+json"})
   GitHub Metadata: Calling @client.releases("trinitronx/action-doctl")
   GitHub Metadata: Calling @client.contributors("trinitronx/action-doctl")
   GitHub Metadata: Calling @client.releases("trinitronx/ansible")
   GitHub Metadata: Calling @client.contributors("trinitronx/ansible")
   GitHub Metadata: Calling @client.releases("trinitronx/ansible-examples")
   GitHub Metadata: Calling @client.contributors("trinitronx/ansible-examples")


[... SNIP ...]
[... TOTAL number of releases + contributors API log lines = 486 ...]
[... SNIP ...]


   GitHub Metadata: Calling @client.releases("trinitronx/zengge-lightcontrol")
   GitHub Metadata: Calling @client.contributors("trinitronx/zengge-lightcontrol")
           Writing: /mnt/terabyte/src/sites/trinitronx.github.io/_site/index.html
  Writing Metadata: .jekyll-metadata
                    done in 177.699 seconds.
 Auto-regeneration: disabled. Use --watch to enable.
[18:31:02] Finished 'jekyll' after 2.97 min

[trinitronx]

Some ideas:

• Simply allow a config option to disable these API calls for sites that don't need releases or contributors data gathered
• Gather the data lazily if the releases or contributors are actually accessed

In case it helps identify the code paths involved for the API calls...

Expand for stacktrace when pressing Ctrl+C during the API call loop

/home/exampleuser/.rvm/rubies/ruby-3.1.4/lib/ruby/3.1.0/net/http.rb:1018:in `initialize': Interrupt
        from /home/exampleuser/.rvm/rubies/ruby-3.1.4/lib/ruby/3.1.0/net/http.rb:1018:in `open'
        from /home/exampleuser/.rvm/rubies/ruby-3.1.4/lib/ruby/3.1.0/net/http.rb:1018:in `block in connect'
        from /home/exampleuser/.rvm/rubies/ruby-3.1.4/lib/ruby/3.1.0/timeout.rb:107:in `block in timeout'
        from /home/exampleuser/.rvm/rubies/ruby-3.1.4/lib/ruby/3.1.0/timeout.rb:117:in `timeout'
        from /home/exampleuser/.rvm/rubies/ruby-3.1.4/lib/ruby/3.1.0/net/http.rb:1016:in `connect'
        from /home/exampleuser/.rvm/rubies/ruby-3.1.4/lib/ruby/3.1.0/net/http.rb:995:in `do_start'
        from /home/exampleuser/.rvm/rubies/ruby-3.1.4/lib/ruby/3.1.0/net/http.rb:984:in `start'
        from /home/exampleuser/.rvm/gems/ruby-3.1.4@gh-metadata/gems/faraday-net_http-3.0.2/lib/faraday/adapter/net_http.rb:112:in `request_with_wrapped_block'
        from /home/exampleuser/.rvm/gems/ruby-3.1.4@gh-metadata/gems/faraday-net_http-3.0.2/lib/faraday/adapter/net_http.rb:102:in `perform_request'
        from /home/exampleuser/.rvm/gems/ruby-3.1.4@gh-metadata/gems/faraday-net_http-3.0.2/lib/faraday/adapter/net_http.rb:66:in `block in call'
        from /home/exampleuser/.rvm/gems/ruby-3.1.4@gh-metadata/gems/faraday-2.7.11/lib/faraday/adapter.rb:45:in `connection'
        from /home/exampleuser/.rvm/gems/ruby-3.1.4@gh-metadata/gems/faraday-net_http-3.0.2/lib/faraday/adapter/net_http.rb:65:in `call'
        from /home/exampleuser/.rvm/gems/ruby-3.1.4@gh-metadata/gems/faraday-2.7.11/lib/faraday/middleware.rb:17:in `call'
        from /home/exampleuser/.rvm/gems/ruby-3.1.4@gh-metadata/gems/faraday-2.7.11/lib/faraday/middleware.rb:17:in `call'
        from /home/exampleuser/.rvm/gems/ruby-3.1.4@gh-metadata/gems/faraday-2.7.11/lib/faraday/middleware.rb:17:in `call'
        from /home/exampleuser/.rvm/gems/ruby-3.1.4@gh-metadata/gems/octokit-6.1.1/lib/octokit/middleware/follow_redirects.rb:73:in `perform_with_redirection'
        from /home/exampleuser/.rvm/gems/ruby-3.1.4@gh-metadata/gems/octokit-6.1.1/lib/octokit/middleware/follow_redirects.rb:61:in `call'
        from /home/exampleuser/.rvm/gems/ruby-3.1.4@gh-metadata/gems/faraday-2.7.11/lib/faraday/rack_builder.rb:153:in `build_response'
        from /home/exampleuser/.rvm/gems/ruby-3.1.4@gh-metadata/gems/faraday-2.7.11/lib/faraday/connection.rb:444:in `run_request'
        from /home/exampleuser/.rvm/gems/ruby-3.1.4@gh-metadata/gems/faraday-2.7.11/lib/faraday/connection.rb:200:in `get'
        from /home/exampleuser/.rvm/gems/ruby-3.1.4@gh-metadata/gems/sawyer-0.9.2/lib/sawyer/agent.rb:99:in `call'
        from /home/exampleuser/.rvm/gems/ruby-3.1.4@gh-metadata/gems/octokit-6.1.1/lib/octokit/connection.rb:156:in `request'
        from /home/exampleuser/.rvm/gems/ruby-3.1.4@gh-metadata/gems/octokit-6.1.1/lib/octokit/connection.rb:84:in `paginate'
        from /home/exampleuser/.rvm/gems/ruby-3.1.4@gh-metadata/gems/octokit-6.1.1/lib/octokit/client/releases.rb:15:in `releases'
        from /home/exampleuser/.rvm/gems/ruby-3.1.4@gh-metadata/gems/jekyll-github-metadata-2.16.0/lib/jekyll-github-metadata/client.rb:63:in `public_send'
        from /home/exampleuser/.rvm/gems/ruby-3.1.4@gh-metadata/gems/jekyll-github-metadata-2.16.0/lib/jekyll-github-metadata/client.rb:63:in `block in method_missing'
        from /home/exampleuser/.rvm/gems/ruby-3.1.4@gh-metadata/gems/jekyll-github-metadata-2.16.0/lib/jekyll-github-metadata/client.rb:92:in `save_from_errors'
        from /home/exampleuser/.rvm/gems/ruby-3.1.4@gh-metadata/gems/jekyll-github-metadata-2.16.0/lib/jekyll-github-metadata/client.rb:63:in `method_missing'
        from /home/exampleuser/.rvm/gems/ruby-3.1.4@gh-metadata/gems/jekyll-github-metadata-2.16.0/lib/jekyll-github-metadata/repository.rb:123:in `block (3 levels) in owner_public_repositories'
        from /home/exampleuser/.rvm/gems/ruby-3.1.4@gh-metadata/gems/jekyll-github-metadata-2.16.0/lib/jekyll-github-metadata/value.rb:53:in `call_or_value'
        from /home/exampleuser/.rvm/gems/ruby-3.1.4@gh-metadata/gems/jekyll-github-metadata-2.16.0/lib/jekyll-github-metadata/value.rb:29:in `render'
        from /home/exampleuser/.rvm/gems/ruby-3.1.4@gh-metadata/gems/jekyll-github-metadata-2.16.0/lib/jekyll-github-metadata/sanitizer.rb:35:in `sanitize'
        from /home/exampleuser/.rvm/gems/ruby-3.1.4@gh-metadata/gems/jekyll-github-metadata-2.16.0/lib/jekyll-github-metadata/sanitizer.rb:55:in `block in sanitize_resource'
        from /home/exampleuser/.rvm/gems/ruby-3.1.4@gh-metadata/gems/jekyll-github-metadata-2.16.0/lib/jekyll-github-metadata/sanitizer.rb:54:in `each'
        from /home/exampleuser/.rvm/gems/ruby-3.1.4@gh-metadata/gems/jekyll-github-metadata-2.16.0/lib/jekyll-github-metadata/sanitizer.rb:54:in `each_with_object'
        from /home/exampleuser/.rvm/gems/ruby-3.1.4@gh-metadata/gems/jekyll-github-metadata-2.16.0/lib/jekyll-github-metadata/sanitizer.rb:54:in `sanitize_resource'
        from /home/exampleuser/.rvm/gems/ruby-3.1.4@gh-metadata/gems/jekyll-github-metadata-2.16.0/lib/jekyll-github-metadata/sanitizer.rb:38:in `sanitize'
        from /home/exampleuser/.rvm/gems/ruby-3.1.4@gh-metadata/gems/jekyll-github-metadata-2.16.0/lib/jekyll-github-metadata/sanitizer.rb:23:in `block in sanitize'
        from /home/exampleuser/.rvm/gems/ruby-3.1.4@gh-metadata/gems/jekyll-github-metadata-2.16.0/lib/jekyll-github-metadata/sanitizer.rb:23:in `map'
        from /home/exampleuser/.rvm/gems/ruby-3.1.4@gh-metadata/gems/jekyll-github-metadata-2.16.0/lib/jekyll-github-metadata/sanitizer.rb:23:in `sanitize'
        from /home/exampleuser/.rvm/gems/ruby-3.1.4@gh-metadata/gems/jekyll-github-metadata-2.16.0/lib/jekyll-github-metadata/value.rb:29:in `render'
        from /home/exampleuser/.rvm/gems/ruby-3.1.4@gh-metadata/gems/jekyll-github-metadata-2.16.0/lib/jekyll-github-metadata/repository.rb:216:in `memoize_value'
        from /home/exampleuser/.rvm/gems/ruby-3.1.4@gh-metadata/gems/jekyll-github-metadata-2.16.0/lib/jekyll-github-metadata/repository.rb:120:in `owner_public_repositories'
        from /home/exampleuser/.rvm/rubies/ruby-3.1.4/lib/ruby/3.1.0/forwardable.rb:238:in `public_repositories'
        from /home/exampleuser/.rvm/gems/ruby-3.1.4@gh-metadata/gems/jekyll-4.3.2/lib/jekyll/drops/drop.rb:128:in `public_send'
        from /home/exampleuser/.rvm/gems/ruby-3.1.4@gh-metadata/gems/jekyll-4.3.2/lib/jekyll/drops/drop.rb:128:in `[]'
        from /home/exampleuser/.rvm/gems/ruby-3.1.4@gh-metadata/gems/liquid-4.0.4/lib/liquid/context.rb:197:in `lookup_and_evaluate'
        from /home/exampleuser/.rvm/gems/ruby-3.1.4@gh-metadata/gems/liquid-4.0.4/lib/liquid/variable_lookup.rb:48:in `block in evaluate'

[aspiers]

I see something similar; it's querying releases and contributors for ALL repos I own, even though I'm only using Jekyll on a single one - why is it doing this?!

[elichad]

Bumping this old issue - I use this plugin only to propagate the variables listed in the README, and have no need for it to query every repository under my username. I regularly run into the API rate limit during development as I have a lot of repos! I would really appreciate an option to disable those API calls when they're not needed.