Auto merge of #2857 - alexcrichton:redirect-sources, r=brson

Add support local mirrors of registries, take 2

This series of commits culminates in first class support in Cargo for local mirrors of registries. This is implemented through a number of other more generic mechanisms, and extra support was added along the way. The highlights of this PR, however, are:

New `.cargo/config` keys have been added to enable *replacing one source with another*. This functionality is intended to be used for mirrors of the main registry or otherwise one to one source correspondences. The support looks like:

```toml
[source.crates-io]
replace-with = 'my-awesome-registry'

[source.my-awesome-registry]
registry = 'https://github.com/my-awesome/registry-index'
```

This configuration means that instead of using `crates-io` (e.g. `https://github.com/rust-lang/crates.io-index`), Cargo will query the `my-awesome-registry` source instead (configured to a different index here). This alternate source **must be the exact same as the crates.io index**. Cargo assumes that replacement sources are exact 1:1 mirrors in this respect, and the following support is designed around that assumption.

When generating a lock file for crate using a replacement registry, the *original registry* will be encoded into the lock file. For example in the configuration above, all lock files will still mention crates.io as the registry that packages originated from. This semantically represents how crates.io is the source of truth for all crates, and this is upheld because all replacements have a 1:1 correspondance.

Overall, this means that no matter what replacement source you're working with, you can ship your lock file to anyone else and you'll all still have verifiably reproducible builds!

With the above support for custom registries, it's now possible for a project to be downloading crates from any number of sources. One of Cargo's core goals is reproducible builds, and with all these new sources of information it may be easy for a few situations to arise:

1. A local replacement of crates.io could be corrupt
2. A local replacement of crates.io could have made subtle changes to crates

In both of these cases, Cargo would today simply give non-reproducible builds. To help assuage these concerns, Cargo will now track the sha256 checksum of all crates from registries in the lock file. Whenever a `Cargo.lock` is generated from now on it will contain a `[metadata]` section which lists the sha256 checksum of all crates in the lock file (or `<none>` if the sha256 checksum isn't known).

Cargo already checks registry checksums against what's actually downloaded, and Cargo will now verify between iterations of the lock file that checksums remain the same as well. This means that if a local replacement registry is **not** in a 1:1 correspondance with crates.io, the lock file will prevent the build from progressing until the discrepancy is resolved.

In addition to the support above, there is now a new kind of source in Cargo, a "local registry", which is intended to be a subset of the crates.io ecosystem purposed for a local build for any particular project here or there. The way to enable this looks like:

```toml
[source.crates-io]
replace-with = 'my-awesome-registry'

[source.my-awesome-registry]
local-registry = 'path/to/my/local/registry'
```

This local registry is expected to have two components:

1. A directory called `index` which matches the same structure as the crates.io index. The `config.json` file is not required here.
2. Inside the registry directory are any number of `.crate` files (downloaded from crates.io). Each crate file has the name `<package>-<version>.crate`.

This local registry must currently be managed manually, but I plan on publishing and maintaining a Cargo subcommand to manage a local registry. It will have options to do things like:

1. Sync a local registry with a `Cargo.lock`
2. Add a registry package to a local registry
3. Remove a package from a local registry

In addition to local registries, Cargo also supports a "directory source" like so

```toml
[source.crates-io]
replace-with = 'my-awesome-registry'

[source.my-awesome-registry]
directory = 'path/to/some/sources'
```

A directory source is similar to a local registry above, except that all the crates are unpacked and visible as vendored source. This format is suitable for checking into source trees, like Gecko's.

Unlike local registries above we don't have a tarball to verify the crates.io checksum with, but each vendored dependency has metadata containing what it *would* have been. To further prevent modifications by accident, the metadata contains the checksum of each file which should prevent accidental local modifications and steer towards `[replace]` as the mechanism to edit dependencies if necessary.

This is quite a bit of new features! What's all this meant to do? Some example scenarios that this is envisioned to solve are:

1. Supporting mirrors for crates.io in a first class fashion. Once we have the ability to spin up your own local registry, it should be easy to locally select a new mirror.
2. Supporting round-robin mirrors, this provides an easy vector for configuration of "instead of crates.io hit the first source in this list that works"
3. Build environments where network access is not an option. Preparing a local registry ahead-of-time (from a known good lock file) will be a vector to ensure that all Rust dependencies are locally available.
   * Note this is intended to include use cases like Debian and Gecko

Even with the new goodies here, there's some more vectors through which this can be expanded:

* Support for running your own mirror of crates.io needs to be implemented to be "easy to do". There should for example be a `cargo install foo` available to have everything "Just Work".
* Replacing a source with a list of sources (attempted in round robin fashion) needs to be implemented
* Eventually this support will be extended to the `Cargo.toml` file itself. For example:
  * packages should be downloadable from multiple registries
  * replacement sources should be encodable into `Cargo.toml` (note that these replacements, unlike the ones above, would be encoded into `Cargo.lock`)
  * adding multiple mirrors to a `Cargo.toml` should be supported
* Implementing the subcommand above to manage local registries needs to happen (I will attend to this shortly)

Author: bors

Makefile.in

@@ -136,7 +136,7 @@ clean:
 # === Documentation
 
 DOCS := index faq config guide manifest build-script pkgid-spec crates-io \
-	environment-variables specifying-dependencies
+	environment-variables specifying-dependencies source-replacement
 DOC_DIR := target/doc
 DOC_OPTS := --markdown-no-toc \
 		--markdown-css stylesheets/normalize.css \

src/bin/git_checkout.rs

@@ -1,6 +1,6 @@
 use cargo::core::source::{Source, SourceId, GitReference};
 use cargo::sources::git::{GitSource};
-use cargo::util::{Config, CliResult, CliError, human, ToUrl};
+use cargo::util::{Config, CliResult, ToUrl};
 
 #[derive(RustcDecodable)]
 pub struct Options {
@@ -37,20 +37,14 @@ pub fn execute(options: Options, config: &Config) -> CliResult<Option<()>> {
                           options.flag_locked));
     let Options { flag_url: url, flag_reference: reference, .. } = options;
 
-    let url = try!(url.to_url().map_err(|e| {
-                       human(format!("The URL `{}` you passed was \
-                                      not a valid URL: {}", url, e))
-                   })
-                   .map_err(|e| CliError::new(e, 1)));
+    let url = try!(url.to_url());
 
     let reference = GitReference::Branch(reference.clone());
     let source_id = SourceId::for_git(&url, reference);
 
     let mut source = GitSource::new(&source_id, config);
 
-    try!(source.update().map_err(|e| {
-        CliError::new(human(format!("Couldn't update {:?}: {:?}", source, e)), 1)
-    }));
+    try!(source.update());
 
     Ok(None)
 }

src/bin/install.rs

@@ -1,6 +1,6 @@
 use cargo::ops;
 use cargo::core::{SourceId, GitReference};
-use cargo::util::{CliResult, Config, ToUrl, human};
+use cargo::util::{CliResult, Config, ToUrl};
 
 #[derive(RustcDecodable)]
 pub struct Options {
@@ -116,7 +116,7 @@ pub fn execute(options: Options, config: &Config) -> CliResult<Option<()>> {
     };
 
     let source = if let Some(url) = options.flag_git {
-        let url = try!(url.to_url().map_err(human));
+        let url = try!(url.to_url());
         let gitref = if let Some(branch) = options.flag_branch {
             GitReference::Branch(branch)
         } else if let Some(tag) = options.flag_tag {
@@ -132,7 +132,7 @@ pub fn execute(options: Options, config: &Config) -> CliResult<Option<()>> {
     } else if options.arg_crate == None {
         try!(SourceId::for_path(&config.cwd()))
     } else {
-        try!(SourceId::for_central(config))
+        try!(SourceId::crates_io(config))
     };
 
     let krate = options.arg_crate.as_ref().map(|s| &s[..]);

src/bin/login.rs

@@ -43,10 +43,10 @@ pub fn execute(options: Options, config: &Config) -> CliResult<Option<()>> {
     let token = match options.arg_token.clone() {
         Some(token) => token,
         None => {
-            let src = try!(SourceId::for_central(config));
-            let mut src = RegistrySource::new(&src, config);
+            let src = try!(SourceId::crates_io(config));
+            let mut src = RegistrySource::remote(&src, config);
             try!(src.update());
-            let config = try!(src.config());
+            let config = try!(src.config()).unwrap();
             let host = options.flag_host.clone().unwrap_or(config.api);
             println!("please visit {}me and paste the API Token below", host);
             let mut line = String::new();

src/cargo/core/dependency.rs

@@ -249,6 +249,7 @@ impl Dependency {
     pub fn is_transitive(&self) -> bool { self.inner.is_transitive() }
     pub fn is_build(&self) -> bool { self.inner.is_build() }
     pub fn is_optional(&self) -> bool { self.inner.is_optional() }
+
     /// Returns true if the default features of the dependency are requested.
     pub fn uses_default_features(&self) -> bool {
         self.inner.uses_default_features()
@@ -263,6 +264,17 @@ impl Dependency {
     pub fn matches_id(&self, id: &PackageId) -> bool {
         self.inner.matches_id(id)
     }
+
+    pub fn map_source(self, to_replace: &SourceId, replace_with: &SourceId)
+                      -> Dependency {
+        if self.source_id() != to_replace {
+            self
+        } else {
+            Rc::try_unwrap(self.inner).unwrap_or_else(|r| (*r).clone())
+               .set_source_id(replace_with.clone())
+               .into_dependency()
+        }
+    }
 }
 
 impl Platform {

src/cargo/core/manifest.rs

@@ -4,7 +4,8 @@ use std::path::{PathBuf, Path};
 use semver::Version;
 use rustc_serialize::{Encoder, Encodable};
 
-use core::{Dependency, PackageId, PackageIdSpec, Summary, WorkspaceConfig};
+use core::{Dependency, PackageId, Summary, SourceId, PackageIdSpec};
+use core::WorkspaceConfig;
 use core::package_id::Metadata;
 
 pub enum EitherManifest {
@@ -232,6 +233,14 @@ impl Manifest {
     pub fn set_summary(&mut self, summary: Summary) {
         self.summary = summary;
     }
+
+    pub fn map_source(self, to_replace: &SourceId, replace_with: &SourceId)
+                      -> Manifest {
+        Manifest {
+            summary: self.summary.map_source(to_replace, replace_with),
+            ..self
+        }
+    }
 }
 
 impl VirtualManifest {

src/cargo/core/package.rs

@@ -99,6 +99,14 @@ impl Package {
                                     .filter(|&(d, _)| d < 4);
         matches.min_by_key(|t| t.0).map(|t| t.1)
     }
+
+    pub fn map_source(self, to_replace: &SourceId, replace_with: &SourceId)
+                      -> Package {
+        Package {
+            manifest: self.manifest.map_source(to_replace, replace_with),
+            manifest_path: self.manifest_path,
+        }
+    }
 }
 
 impl fmt::Display for Package {

src/cargo/core/package_id.rs

@@ -13,12 +13,12 @@ use util::{CargoResult, CargoError, short_hash, ToSemver};
 use core::source::SourceId;
 
 /// Identifier for a specific version of a package in a specific source.
-#[derive(Clone, Debug)]
+#[derive(Clone)]
 pub struct PackageId {
     inner: Arc<PackageIdInner>,
 }
 
-#[derive(PartialEq, PartialOrd, Eq, Ord, Debug)]
+#[derive(PartialEq, PartialOrd, Eq, Ord)]
 struct PackageIdInner {
     name: String,
     version: semver::Version,
@@ -38,13 +38,19 @@ impl Decodable for PackageId {
     fn decode<D: Decoder>(d: &mut D) -> Result<PackageId, D::Error> {
         let string: String = try!(Decodable::decode(d));
         let regex = Regex::new(r"^([^ ]+) ([^ ]+) \(([^\)]+)\)$").unwrap();
-        let captures = regex.captures(&string).expect("invalid serialized PackageId");
+        let captures = try!(regex.captures(&string).ok_or_else(|| {
+            d.error("invalid serialized PackageId")
+        }));
 
         let name = captures.at(1).unwrap();
         let version = captures.at(2).unwrap();
         let url = captures.at(3).unwrap();
-        let version = semver::Version::parse(version).ok().expect("invalid version");
-        let source_id = SourceId::from_url(url);
+        let version = try!(semver::Version::parse(version).map_err(|_| {
+            d.error("invalid version")
+        }));
+        let source_id = try!(SourceId::from_url(url).map_err(|e| {
+            d.error(&e.to_string())
+        }));
 
         Ok(PackageId {
             inner: Arc::new(PackageIdInner {
@@ -151,6 +157,16 @@ impl PackageId {
             }),
         }
     }
+
+    pub fn with_source_id(&self, source: &SourceId) -> PackageId {
+        PackageId {
+            inner: Arc::new(PackageIdInner {
+                name: self.inner.name.to_string(),
+                version: self.inner.version.clone(),
+                source_id: source.clone(),
+            }),
+        }
+    }
 }
 
 impl Metadata {
@@ -173,16 +189,26 @@ impl fmt::Display for PackageId {
     }
 }
 
+impl fmt::Debug for PackageId {
+    fn fmt(&self, f: &mut Formatter) -> fmt::Result {
+        f.debug_struct("PackageId")
+         .field("name", &self.inner.name)
+         .field("version", &self.inner.version.to_string())
+         .field("source", &self.inner.source_id.to_string())
+         .finish()
+    }
+}
+
 #[cfg(test)]
 mod tests {
     use super::PackageId;
     use core::source::SourceId;
-    use sources::RegistrySource;
+    use sources::CRATES_IO;
     use util::ToUrl;
 
     #[test]
     fn invalid_version_handled_nicely() {
-        let loc = RegistrySource::default_url().to_url().unwrap();
+        let loc = CRATES_IO.to_url().unwrap();
         let repo = SourceId::for_registry(&loc);
 
         assert!(PackageId::new("foo", "1.0", &repo).is_err());

src/cargo/core/registry.rs

@@ -3,13 +3,22 @@ use std::collections::{HashSet, HashMap};
 use core::{Source, SourceId, SourceMap, Summary, Dependency, PackageId, Package};
 use core::PackageSet;
 use util::{CargoResult, ChainError, Config, human, profile};
+use sources::config::SourceConfigMap;
 
 /// Source of information about a group of packages.
 ///
 /// See also `core::Source`.
 pub trait Registry {
     /// Attempt to find the packages that match a dependency request.
     fn query(&mut self, name: &Dependency) -> CargoResult<Vec<Summary>>;
+
+    /// Returns whether or not this registry will return summaries with
+    /// checksums listed.
+    ///
+    /// By default, registries do not support checksums.
+    fn supports_checksums(&self) -> bool {
+        false
+    }
 }
 
 impl Registry for Vec<Summary> {
@@ -26,6 +35,12 @@ impl Registry for Vec<Package> {
     }
 }
 
+impl<'a, T: ?Sized + Registry + 'a> Registry for Box<T> {
+    fn query(&mut self, name: &Dependency) -> CargoResult<Vec<Summary>> {
+        (**self).query(name)
+    }
+}
+
 /// This structure represents a registry of known packages. It internally
 /// contains a number of `Box<Source>` instances which are used to load a
 /// `Package` from.
@@ -41,7 +56,6 @@ impl Registry for Vec<Package> {
 /// operations if necessary) and is ready to be queried for packages.
 pub struct PackageRegistry<'cfg> {
     sources: SourceMap<'cfg>,
-    config: &'cfg Config,
 
     // A list of sources which are considered "overrides" which take precedent
     // when querying for packages.
@@ -65,6 +79,7 @@ pub struct PackageRegistry<'cfg> {
     source_ids: HashMap<SourceId, (SourceId, Kind)>,
 
     locked: HashMap<SourceId, HashMap<String, Vec<(PackageId, Vec<PackageId>)>>>,
+    source_config: SourceConfigMap<'cfg>,
 }
 
 #[derive(PartialEq, Eq, Clone, Copy)]
@@ -75,14 +90,15 @@ enum Kind {
 }
 
 impl<'cfg> PackageRegistry<'cfg> {
-    pub fn new(config: &'cfg Config) -> PackageRegistry<'cfg> {
-        PackageRegistry {
+    pub fn new(config: &'cfg Config) -> CargoResult<PackageRegistry<'cfg>> {
+        let source_config = try!(SourceConfigMap::new(config));
+        Ok(PackageRegistry {
             sources: SourceMap::new(),
             source_ids: HashMap::new(),
             overrides: Vec::new(),
-            config: config,
+            source_config: source_config,
             locked: HashMap::new(),
-        }
+        })
     }
 
     pub fn get(self, package_ids: &[PackageId]) -> PackageSet<'cfg> {
@@ -158,8 +174,8 @@ impl<'cfg> PackageRegistry<'cfg> {
 
     fn load(&mut self, source_id: &SourceId, kind: Kind) -> CargoResult<()> {
         (|| {
-            // Save off the source
-            let source = source_id.load(self.config);
+            let source = try!(self.source_config.load(source_id));
+
             if kind == Kind::Override {
                 self.overrides.push(source_id.clone());
             }