How to use DataLoader?

[zuohuadong]

Are there any relevant examples?

https://github.com/facebook/dataloader

[Mirokko]

You probably wanna start with Interceptors.

[Mirokko]

@twilroad Nope, but link above is basically everything you need to get started. Interceptor can be used for caching (you can see it in docs), but with slight modifications you can store result from next handler. As a param, you will need a cache key, you can pass it like shown here https://docs.nestjs.com/advanced/mixins.

[caseyduquettesc]

nestjs/graphql#2 (comment)

[babbarankit]

Sample Code

export class ApplicationModule implements NestModule {
  constructor(
    private readonly articleService: ArticleService,
    private readonly graphQLFactory: GraphQLFactory,
    private readonly userService: UserService,
    private readonly cinemaClassicsService: CinemaClassicsService,
    private readonly artformCategoryService: ArtformCategoryService,
    private readonly configService: ConfigService,
    private readonly geoChapterService: GeoChapterService,
    private readonly downloadCategoryService: DownloadCategoryService,
    private readonly artformService: ArtformService,
    private readonly personalProfileService: PersonalProfileService,
    private readonly artistProfileService: ArtistProfileService,
    private readonly artistListService: ArtistListService,
  ) {}
  configure(consumer: MiddlewareConsumer) {
    const awsSettings = this.configService.getAWSConfig();
    const disableCors = this.configService.disableCors();
    let headers: {
      'Access-Control-Allow-Origin'?: string;
    } = {};
    if (disableCors) {
      headers = { 'Access-Control-Allow-Origin': 'http://localhost:3010' };
    }
    consumer
      .apply(
        require('react-s3-uploader/s3router')({
          bucket: awsSettings.defaultBucket,
          region: awsSettings.defaultRegion,
          signatureVersion: 'v4',
          headers,
          ACL: 'public-read',
          uniquePrefix: true,
        }),
      )
      .forRoutes('/s3');
    consumer.apply(AuthMiddleware).forRoutes('/health/anonymous');

    consumer.apply(AuthMiddleware).forRoutes('/health/authenticated');

    consumer.apply(AuthMiddleware).forRoutes('/query');

    const typeDefs = this.graphQLFactory.mergeTypesByPaths('./**/*.graphql', './../node_modules/@ankitbabbar/instiapp-api-core/src/**/*.graphql');
    const resolvers = {
      DateTime: GraphQLDateTime,
      Date: GraphQLDate,
      Time: GraphQLTime,
    };

    const schema = this.graphQLFactory.createSchema({
      typeDefs,
      resolvers,
      resolverValidationOptions: {
        requireResolversForResolveType: false,
      },
    });

    consumer
      .apply(
        graphqlExpress((req, res) => {
          const viewer = res.locals.viewer;
          const dataLoaders: ContribDataLoaders = {
            user: new DataLoader((ids: number[]) => this.userService.getManyByIds(viewer, ids)),
            article: new DataLoader((ids: number[]) => this.articleService.getManyByIds(viewer, ids)),
            cinemaClassics: new DataLoader((ids: number[]) => this.cinemaClassicsService.getManyByIds(viewer, ids)),
            artform: new DataLoader((ids: number[]) => this.artformService.getManyByIds(viewer, ids)),
            geoChapter: new DataLoader((ids: number[]) => this.geoChapterService.getManyByIds(viewer, ids)),
            artformCategory: new DataLoader((ids: number[]) => this.artformCategoryService.getManyByIds(viewer, ids)),
            downloadCategory: new DataLoader((ids: number[]) => this.downloadCategoryService.getManyByIds(viewer, ids)),
            personalProfile: new DataLoader((ids: number[]) => this.personalProfileService.getManyByIds(viewer, ids)),
            artistList: new DataLoader((ids: number[]) => this.artistListService.getManyByIds(viewer, ids)),
            artistProfile: new DataLoader((ids: number[]) => this.artistProfileService.getManyByIds(viewer, ids)),
          };
          const context: IGQLContext = {
            viewer,
            dataLoaders,
          };
          return {
            schema,
            rootValue: req,
            formatError,
            context,
          };
        }),
      )
      .forRoutes('/query');
  }
}

[biels]

How would this work with the new GraphQLModule.forRoot({...options})? Where should we initialize the context? Maybe creating a wrapper module and doing it on configure(consumer) somehow? It should be a different instance of data loader(s) for each different request.

[mohaalak]

How would this work with the new GraphQLModule.forRoot({...options})? Where should we initialize the context? Maybe creating a wrapper module and doing it on configure(consumer) somehow? It should be a different instance of data loader(s) for each different request.

take a look at this approach
nestjs/graphql#2 (comment)

[rychkog]

@kamilmysliwiec Dataloader is supposed to batch all the resolved entity ids and query them in one go as a single SQL query.

But for some reason when used in Nest it executes more than one query for a single request.

Here is an example

Suppose we have a following GQL query

query {
  persons {
    name
    address {
      street
    }
  }
}

So addresses for all the persons from this query should be fetched using a single SQL query (using dataloader approach) but in reality, I have something like this:

1. All the persons fetched
2. Dataloader collects and loads address ids for persons 1 and 2 (SQL query 1)
3. Then for person 3 (SQL query 2)
4. Then for persons 4 and 5 (SQL query 3)

I know that dataloader should be invoked on a nextTick but looks like there is something in a way NestJS implements resolvers.

Any help is appreciated, maybe I just missed something regarding the dataloader..

I think this is related graphql/dataloader#150

[rychkog]

For those who faced the same issue, I've mentioned above.
It was caused by the fact that on every GQL property resolution Nest invokes my Auth global guard which also does lots of async stuff, interfering with dataloader's execution.

I've added a flag that checks whether a user is already authenticated in the guard in order to prevent its redundant executions.

@kamilmysliwiec Is this expected behavior that guards (probably interceptors, filters, etc. as well) get executed on every GQL property resolution?

[kamilmysliwiec]

@kamilmysliwiec Is this expected behavior that guards (probably interceptors, filters, etc. as well) get executed on every GQL property resolution?

@rychkog this is fixed in @nestjs/graphql@next already. I'll publish this release as soon as possible.

[jakubnavratil]

@kamilmysliwiec maybe this could be somewhat optional? you disabled all guards, filters and interceptors. what if I want to run guard againts some property?

In my case I was running interceptor altering the context and this was breaking change. I think this ability could be actually in ResolveProperty options or something, especially guards should be allowed.

[kamilmysliwiec]

@trubit it would have a big impact on the performance. I'd suggest moving this logic inside the resolve property handler.

[EdouardBougon]

I agree with @trubit , it's a breaking change.

For example, I use guard to control access to my properties, with a AuthGuard and RoleGuard.
Some properties are like "query", with filter, pagination etc... and need the same rules as query or mutation.

So now, we will have 2 different implementations to check roles and auth ...

[jakubnavratil]

I imagine this line
https://github.com/nestjs/graphql/blob/1504e021af3119feb8d8ecaed29b47d695939d9c/lib/services/resolvers-explorer.service.ts#L134
could be somehow referenced in @ResolveProperty's decorator options. At least making it possible to add guards on properties. Of course it takes some performance hits, was it so big?

[vnenkpet]

@zuohuadong How I do it is I use request-scoped providers that create instances of injectable services that have dataloaders in them.

export interface IDataLoader<K, V> {
  load(id: K): Promise<V>;
}

import DataLoader from 'dataloader';
import { UserEntity } from '../user.entity';
import { IDataLoader } from 'src/api/common/interfaces/data-loader.interface';
import { Connection } from 'typeorm';

export class UsersDataLoader implements IDataLoader<string, UserEntity> {
  constructor(private readonly dataLoader: DataLoader<string, UserEntity>) {}

  public static async create(connection: Connection): Promise<UsersDataLoader> {
    const usersRepo = connection.getRepository<UserEntity>(UserEntity);
    const dataLoader = new DataLoader<string, UserEntity>(async keys => {
      const loadedEntities = await usersRepo.findByIds(keys);
      return keys.map(key => loadedEntities.find(entity => entity.id === key)); // sort by keys
    });

    return new UsersDataLoader(dataLoader);
  }

  public async load(id: string) {
    return this.dataLoader.load(id);
  }
}

import { Provider, Scope } from '@nestjs/common';
import { Types } from 'src/common/types';
import { UsersDataLoader } from './users.data-loader';

export const usersDataLoaderProvider: Provider = {
  inject: [Types.DATABASE_CONNECTION],
  useFactory: UsersDataLoader.create,
  provide: UsersDataLoader,
  scope: Scope.REQUEST,
};

Then you simply inject it as a regular service in the resolver.

@Resolver(of => PostType)
export class PostResolver {
  constructor(
    private readonly usersDataLoader: UsersDataLoader,
  ) {}
  @ResolveProperty('author')
  public async author(@Parent() post: PostEntity) {
    return this.usersDataLoader.load(post.authorId);
  }
}

This way you can also use the dataloaders anywhere in the code, like in service classes, not only in resolver methods.