Creating a Scala.js Open Source Project

The Scala community in general (and the Scala.js community in particular) are highly dependent on open-source libraries -- without even Typesafe backing us officially, we need to pool our resources and share as much as is feasible. I wanted to join in: Querki is open-but-proprietary (it is intended to be a serious business), but I've built lots of useful bits and pieces along the way and wanted to spin them out into proper open-source libraries. But when I went to do so, I was just plain intimidated for a few months, and even when I got pointers from folks, it wasn't easy to figure out the correct order of events from the thicket of documentation.
This page tries to help with that: I've taken the various instructions I found, and my experiences building my first couple of libraries, and turned them into a step-by-step guide for creating an open-source library. It is intentionally pretty detailed, and some folks will find many of the steps obvious, but I'd rather be too complete than leave things out. It's a bit opinionated, and oriented towards how I do things; obviously, change whatever details don't work for your process. It is specific to releasing via Sonatype, since that seems to be the favorite platform among the Scala.js community. I hope you find it useful.
These instructions are specific to creating a Scala.js library, since that is what I've been doing. I believe that the procedures for creating a Scala-JVM library or a cross-compiled one are very similar, differing mainly in the setup of your build.sbt file. (None of which is terribly specific to releasing the library, just to creating it in the first place.)
These instructions are focused on refactoring a library out of an existing code base, since that's the way I typically operate -- I like to get something working in Querki first, then extract it if it seems robust and useful. If you are building a library from scratch, the process should be similar, but you'll probably want to create a test project to make sure it's all working.
Steps that are marked (Only once) only need to happen once: they are for getting your organization or computer primed for creating open-source libraries. Subsequent libraries do not necessarily need to repeat these steps.

Resources

The following pages proved particularly useful to me while figuring this all out; the Instructions below are mostly adapted from them:

Preconditions

I assume that you already understand at least the basics of git, Scala and sbt. You don't have to be expert in any of them, but I'm not going to explain the core concepts of them.
I assume that you have sbt or Activator installed. It doesn't matter which -- for our purposes, they're effectively identical.

Instructions

(Only once) Create a GitHub account: If you don't already have an account at GitHub.com, create one. (Really, if you don't already have an account, I recommend creating one and spending a while using and contributing to other open-source projects first. There's an ethos and style surrounding the open-source community, and it's not all obvious if you are new to this world.)
Create a GitHub repo: From the front page of GitHub, create a New Repository. Give it an appropriate name. I recommend letting GitHub create an initial README.md, which is what will show up on the GitHub page and should be your core documentation. (This file is in Markdown format; if you don't know Markdown, I recommend familiarizing yourself with its syntax.)
Clone the empty repo to your development machine
Copy the relevant code from the main project: As I said, I'm usually extracting already-working libraries from Querki. So in the local repo, I create the path to where the source code belongs (for org.querki libraries, the path is src/main/scala/org/querki/library), and copy the actual library code into there. Your path probably wants to be something similar, possibly tweaked for more complex projects.
Put a bit of skeleton into README.md: Not required, but I think it's good practice to at least briefly describe what the project will do, and put the license into there, so you don't forget it.
Commit and sync the code to GitHub: Note that it doesn't have to work yet, but it's good to have the skeleton in place before the next step.
(Only once) Request a GroupID from Sonatype: Actually, you're asking them to set things up for your initial "artifact", the library you are developing, but the important part is that they create your organization so you have somewhere to upload to. Go to this page and fill in the form:
  • The Description isn't required, although I like to provide a brief paragraph of what I'm doing here.
  • The most important field is the Group ID, which is similar to a typical Java or Scala namespace. The full details are given here, but suffice it to say that Querki uses "org.querki", the same as the namespace for the library itself.
  • Fill in the Project URL as the homepage if you have one; otherwise, just give the URL of your GitHub repo.
  • The SCM URL is the URL of your GitHub repo, plus ".git" at the end.
Note that, after you press the "Create" button, it'll take a little while -- this step involves some human intervention, so be prepared to wait at least a few hours. You should get an email promptly telling you that you have created a JIRA Issue, but that is not relevant: you need to wait until you get an email saying that this issue is resolved before you try to release your library.
After you have released your first library, you don't need to do this step again: Sonatype only requires this manual intervention for your first "artifact".
Create the project directory: while waiting for Sonatype, get things actually working. Create a project/ directory at the top level of your local repo.
Create project/build.properties: this is a standard file, and just says which version of sbt to use:
sbt.version=0.13.16
Create project/build.sbt: this is where you put your sbt plugins. The important two lines are:
addSbtPlugin("org.scala-js" % "sbt-scalajs" % "0.6.11")
addSbtPlugin("com.jsuereth" % "sbt-pgp" % "1.0.0")
The first line says which version of Scala.js you're using. The second says to includes the sbt-pgp plugin, which will make it easier to sign the library for Sonatype.
I also recommend adding:
addSbtPlugin("org.xerial.sbt" % "sbt-sonatype" % "1.1")
This allows you to use the sbt-sonatype plugin later during the release, which makes things easier.
Create the main build.sbt file: in your root directory, define the main build.sbt. Obviously, the details will vary a lot depending on what you library is trying to do, but here's a rough template to start with, including all the critical bits. Replace all the bits in square brackets with your own content; I've included the examples from jsext in comments:
import SonatypeKeys._

sonatypeSettings

lazy val root = project.in(file(".")).
  enablePlugins(ScalaJSPlugin)

//name := "jsext Library for Scala.js"
name := "[brief description of your library]"

//normalizedName := "querki-jsext"
normalizedName := "[name of the library in Maven]"

// -SNAPSHOT means "this release is in development"
version := "0.1-SNAPSHOT"

//organization := "org.querki"
organization := "[the group ID you gave to Sonatype]"

scalaVersion := "2.12.2"

crossScalaVersions := Seq("2.10.6", "2.11.11, 2.12.2")

//homepage := Some(url("http://www.querki.net/"))
homepage := Some(url("[your homepage]"))

// The URL of whichever license you want to use for this library:
licenses += ("MIT License", url("http://www.opensource.org/licenses/mit-license.php"))

//scmInfo := Some(ScmInfo(
//    url("https://github.com/jducoeur/jsext"),
//    "scm:git:git@github.com/jducoeur/jsext.git",
//    Some("scm:git:git@github.com/jducoeur/jsext.git")))
scmInfo := Some(ScmInfo(
    url("https://github.com/[your login]/[repo]"),
    "scm:git:git@github.com/[your login]/[repo].git",
    Some("scm:git:git@github.com/[your login]/[repo].git")))

publishMavenStyle := true

publishTo := {
  val nexus = "https://oss.sonatype.org/"
  if (isSnapshot.value)
    Some("snapshots" at nexus + "content/repositories/snapshots")
  else
    Some("releases" at nexus + "service/local/staging/deploy/maven2")
}

//pomExtra := (
//  <developers>
//    <developer>
//      <id>jducoeur</id>
//      <name>Mark Waks</name>
//      <url>https://github.com/jducoeur/</url>
//    </developer>
//  </developers>
//)
pomExtra := (
  <developers>
    <developer>
      <id>[your Sonatype id]</id>
      <name>[your name]</name>
      <url>[your personal url]</url>
    </developer>
  </developers>
)

pomIncludeRepository := { _ => false }
Publish the library locally: In your library's directory, run sbt. Compile, and make sure it seems to be compiling correctly. Once it is, say publishLocal. This publishes the library to your local Ivy cache, which is generally located under ~/.ivy2/local. You might want to sanity-check that the expected pieces got put there -- there should be a subdirectory with the name of your Group ID, with some nested subdirectories that include:
  • docs/ -- has a jar containing the compiled Scaladoc for your library. (You did define some Scaladoc comments, right? If not, think about doing so.)
  • jars/ -- has a jar containing the compiled class files.
  • poms/ -- has a .pom file that describes the project for Sonatype. This should reflect what you described in your build.sbt.
  • srcs/ -- has a jar containing the source code of the library.
If you want to inspect the jars, and don't have a proper jar inspector, remember that .jar is basically the same as .zip -- you can typically copy the file, rename it to .zip and open it that way.
Switch to using the library from your main code: Assuming that you are refactoring the library out of a main project, the way I am, make a git branch of the main project. In that branch, delete the code that became the library, and confirm that it no longer compiles. Then, in the main project's build.sbt, add the library to the libraryDependencies, like this:
"org.querki" %%% "querki-jsext" % "0.1-SNAPSHOT"
Refresh sbt/Activator, and confirm that everything now compiles again, and runs properly. If it does, you have a working library. Congrats -- now you just have to release it.
(Once only) Set up your public/private keypair: Sonatype requires that your library be properly signed, with a publicly available keypair. This is why we included the sbt-pgp plugin earlier. Go back to the library in sbt/Activator. Say set pgpReadOnly := false, which allows you to rewrite your keypairs. Say pgp-cmd gen-key. This should prompt you for several things:
  • Set name associated with the project: (Your company or organization; mine is Querki Inc.)
  • Set email associated with the key:
  • Enter passphrase: (I recommend generating something nice and strong, and keeping it in a password vault)
  • Re-enter passphrase:
Having done that, sbt-pgp will click and whirr for a little while, and spit out a new keypair in ~/.sbt/gpg/. This consists of pubring.asc (the public key) and secring.asc (the private key). Copy both to somewhere safe and secure -- remember to keep the private key highly confidential!
Next, say pgp-cmd send-key [email address] hkp://pool.sks-keyservers.net to publish the public key to an accessible location. Note that this needs the name, email or hex key id of the key, to identify what it's supposed to publish.
Define a sonatype.sbt file, as described in the fourth section of this page. This conventionally gets placed in ~/.sbt/0.13/sonatype.sbt, and contains:
credentials += Credentials("Sonatype Nexus Repository Manager",
                           "oss.sonatype.org",
                           "<your username>",
                           "<your password>")
This tells sbt how to publish to Sonatype. IMPORTANT: note that none of these files under ~.sbt get checked into git. That's intentional -- they contain highly sensitive information! If you move to another computer, you will need to copy these files to there.
Prep for Release: In your build.sbt file, change the version to 0.1 (that is, remove the "-SNAPSHOT"), and recompile.
Publish the signed artifacts: In sbt, say +publishSigned. (The + tells sbt to publish for all the specified Scala versions.) This invokes sbt-pgp to use those digital signatures you created in order to sign all the materials, and then pushes them up to Sonatype. It should take a minute or two. Along the way, it will ask you to re-enter the passphrase you created for your keypair.
Examine and Release the artifacts: Now we're following the steps described in Sonatype's Releasing the Deployment guide. To do this manually:
  • Go to the Sonatype homepage and Log In.
  • Click on Staging Repositories in the left-hand menu. This will display a list of repositories; look for the one whose name is based on your Group Id. (So for example, the first release under org.querki was named "orgquerki-1000".) Click on that, and the lower pane should show you information about the artifacts you are releasing.
  • In the right-hand side of the lower pane, go to the Content tab, and sanity-check the contents. (This is optional, but I like to make sure things actually built and published correctly.)
  • Press the Close button at the top of the window, and enter a description of the release.
  • Wait a bit, then Refresh once in a while until it shows as Closed. (This takes a minute or two, in my experience.)
  • Once that is done, press the Release button at the top of the page, and enter a description of the release.
  • Wait a while -- eventually a Nexus: Promotion Completed email should show up.
However, if you added the sbt-sonatype plugin earlier, you should be able to skip all of those, and instead just say sonatypeRelease. I've gradually come to only do the manual process when I'm concerned that something is weird; most of the time, sbt-sonatype works nicely and pretty quickly. (But keep in mind that it can still take a minute or three.)
(Once only) Tell Sonatype that you're ready to roll: Now, go back to the JIRA ticket that you submitted at the beginning, asking them to create your repository. Add a comment to that ticket, telling them that you have promoted your first release. There will once again be a modest delay (typically a few hours) for manual intervention, while they sanity-check that you know what you're doing. Once they've done that, they will flip the switch to turned on sync'ing of your repository's releases to the public repos.
Again, this only needs to be done once -- after they've turned things on, publishing and sync'ing should usually happen automatically. Sonatype says to expect it to typically take up to ten minutes, and up to a couple of hours before it shows up in central Search. (Search is usually not crucial to your life, though, so don't worry about that too much.)
Test your release: Go to your local Ivy cache (~/.ivy2) and delete the local subdirectory for your library. Go to the main project (or test project) and confirm that it no longer builds, since it doesn't have the SNAPSHOT library that it's pointing to.
Edit the main project's build.sbt file, changing the version in the libraryDependency for your library. (That is, remove the "-SNAPSHOT".) Reload sbt, recompile, and confirm that everything builds and runs correctly. If it does, congratulations -- you're all set!
List your library publicly: Once the library seems to be in good shape -- you are reasonably convinced that it is ready for other people in the Scala.js community to use it -- submit it for addition to the public Scala.js site by submitting a pull request on the main index page. Please look over that page carefully, and list your library in the most appropriate section.

Maintenance

In order to make changes to your library, I believe this is the full process:
Update your version number: In the library's build.sbt, change the version number to "0.2-SNAPSHOT" (or whatever the appropriate version-in-process is).
Make your code changes
Repeat these steps from above:
  • Publish the library locally
  • Switch to using the library from your main code
  • Prep for Release
  • Publish the signed artifacts
  • Examine and Release the artifacts (maybe using sbt-sonatype)
  • Test your Release

Conclusion

That's the whole process, I believe. Adapt as necessary to your own circumstances, and I do recommend checking out the above-linked pages. Please ping me if you have comments or corrections.