Download resource

[SteveL-MSFT]

Summary of the new feature / enhancement

A resource that allows downloading file(s) from web. Would need security aspects including authn and maybe even validating the server. Proxy might be needed as well.

Proposed technical implementation details (optional)

No response

[michaeltlombardi]

Possible resource instance schema (non-exclusive)

type: object
required: [sourceUri, targetFolder]
dependentRequired:
  clientKeyPath: [clientCertPath]
properties:
  sourceUri:
    description: The URI to download the file from.
    type: string
    format: uri
    writeOnly: true
  targetDirectory:
    description: The folder to download the file to.
    type: string
    format: directoryPath
  targetFileName:
    description: >-
      The name of the file after downloading. Defaults to
      the base name of the URI on the remote server.
    type: string
  checksum:
    description: >-
      Defines the checksum algorithm and value to validate
      the downloaded file against, raising an error if the
      checksums don't match.
    type: object
    minProperties: 1
    maxProperties: 1
    properties:
      sha256: { type: string }
      sha512: { type: string }
      # others as appropriate
  clientCertPath:
    description: >-
      Defines the path to the PEM-formatted certificate to
      use for SSL client authentication. If the certificate
      includes the key, the `clientKeyPath` property isn't
      required. If the certificate doesn't include the key
      and the `clientKeyPath` property isn't specified, the
      resource raises an error.
    type: string
    format: filePath
    writeOnly: true
  clientKeyPath:
    description: >-
      Defines the path to the PEM-formatted file containing the
      private key to use for SSL client authentication. This
      property requires the `clientCertPath` property. If the
      certificate specified by the `clientCertPath` property
      includes the key, this property isn't required.
    type: string
    format: filePath
    writeOnly: true
  headers:
    description: >-
      Defines a set of custom HTTP headers for the HTTP request by
      key-value for each property of this object.
    type: object
    writeOnly: true
  httpAgent:
    description: >-
      Defines the user agent header to identify the HTTP request.
      Defaults to `dsc-resource-download/1.0`.
    type: string
    default: dsc-resource-download
    writeOnly: true
  timeOutSeconds:
    description: >-
      Defines the timeout for the HTTP get operation in seconds.
      Defaults to `600` seconds. A timeout of `0` disables the
      timeout check.
    type: integer
    minimum: 0
    maximum: 65535
    default: 600
    writeOnly: true
  _clobber:
    $ref: https://aka.ms/dsc/canonical_properties/clobber.json

[SteveL-MSFT]

I think initially we just need sufficient and secure capability for known scenarios and not end up with curl as a resource

[michaeltlombardi]

I do think that this resource is a useful example of one that could benefit from Proposal: Extended Resources (#611) - especially for setting the attributes, mode, group, etc for the downloaded file, without requiring a user to specify a series of resources or fully reimplement the functionality of multiple resources in this one.

Edit: It's also a resource that other resources would benefit from extending (e.g. specific handling for downloading and munging some file or just providing useful defaults/friendly interface over a specific download).

For example, I might want to download, extract, and define the permissions/group for a file:

resources:
  - name: Get foo from archive
    type: Microsoft.Dsc/Download
    properties:
      sourceUri: https://archive.tailspintoys.com/foo.tar.gz
      targetDirectory: /opt/tools
      expand: true
      mode: 0755
      group: myGroup

I think the MVP for this resource does not include convenience features like expanding archives or managing the file permissions/attributes/etc, only the download operation. Any improvements can be iteratively delivered (and those resources themselves don't exist yet).

[SteveL-MSFT]

100% agree that expanding an archive makes sense without the need to use a separate resource since it would be pretty common operation. This probably also means by default if expanded the archive file is deleted (we can add a property to explicitly preserve it if there's a real use case for that). Then there's the question of whether we need to validate the hash of the file (which would have been the responsibility of a File resource). The permissions is also a bit troublesome since it has two challenges:

1. permissions being different between Windows and non-Windows
2. it would have also been the responsibility of a File resource

I wonder if perhaps one way to reconcile the overlap between File and Download is to simply encapsulate the download ability into the File resource which would also handle hash validation and permissions. This also means the File resource would handle copy type operations whether the source is http, smb, etc...