Implementing Dasein Cloud
Implementing Dasein Cloud is the process of building your own implementation of the dasein-cloud-core classes to support interaction with a new cloud. This document describes how to get started with that process and what you need to do to complete it.
- Create a clone of https://github.com/greese/dasein-cloud-skeleton
- Alter the project information
- Rename the packages and class names so they are no longer have Skeleton and skeleton in the name.
- Create the Context Test
- Abstract cloud interaction (CloudMethod.java) as is appropriate to your cloud's protocol and API
- Properly implement the DataCenterServices (Geography.java) for your cloud
- Implement all other services
- Test your implementation
The best way to do this is to create a copy dasein-cloud-skeleton into your own Github account and work on it there. You don't really want to fork dasein-cloud-skeleton since you aren't actually forking it, but instead creating a new project. If at some later point you want to contribute it back to Dasein Cloud, we can handle than when you're ready.
If you don't want to do that, you can of course download the skeleton to your local machine and begin working on it there.
Start with the pom.xml file. You will obviously want to do a global search for 'skeleton' and replace it with the name of your cloud. You will also want to edit the contributor list so it has you in it. If you do not intend to contribute the code back to Dasein Cloud, you will also want to edit the copyright information in the pom.xml and src/etc/header.txt class as well as remove the LICENSE-APACHE.txt file and insert your own.
The pom.xml includes some basic dependencies common for most REST-based APIs. In addition, downstream dependencies include support for both JSON and XML parsing. If you intend to contribute back to Dasein Cloud, please do not introduce different JSON or XML parsing libraries.
The skeleton project includes packages in src/main and src/test for org.dasein.cloud.skeleton. Those directories should be renamed and the Java classes under those directories starting with the name. All package references within the sample Java classes should also be changed. You may optionally rename the classes (like MyCloud.java) to something more descriptive.
A good Dasein Cloud implementation does API tracing and wire logging of all API interaction. In fact, if you plan to contribute back to Dasein Cloud, it's a requirement for any implementations contributed back to Dasein Cloud. Doing this enables any tools consuming your implementation to trouble shoot chattiness and wire troubles. It's therefore best to abstract out all cloud interaction into a single call that does API tracing and wire logging.
There's a skeleton for this class called RESTMethod.java. It is structured to implement a RESTful API that may be consumed by client classes. Most APIs are not truly RESTful. Furthermore, if the underlying cloud API is SOAP, this skeleton is completely useless. Other than SOAP APIs, however, it's a good starting point that you can use for abstraction.
The critical elements are the calls to the wire logger and APITrace. Make sure you do the following:
- Log any request you send, along with headers and non-binary payload are logged to the wire logger
- Call APITrace.trace() immediately before any network call
- Log the response, including status line, headers, and response body to the wire logger
If it's a SOAP API, you obviously cannot log the wire information. But you can still call APITrace.trace(). Chances are that the underlying API library (like httpclient) is also logging wire information separately. That's OK. There is value in both sets of logging.
The most basic function in a Dasein Cloud implementation is the testContext() method in the provider object (MyCloud.java in the skeleton project). It does two things:
- Verifies the configured API access credentials
- Returns a canonical account number for the connection
The first job is fairly straightforward: make sure that the authentication credentials provided in the provider context are valid. If they are valid, proceed to the second step. If not, return null. Note that this method NEVER throws an exception. It simply returns null. Null should be interpreted as "for whatever reason, we can't authenticate using these credentials".
The second part is a bit odd. In some cases, the account number a user can see differs from the underlying account number the cloud provider uses to signify resource ownership. This is bizarre behavior and I don't understand why such clouds are so obtuse. But it happens. The testContext method is a mechanism for identifying this distinction. In most cases, you just return the account number the user provides because it is the canonical account number. In the weird clouds, you return what is the "real" account number used in API calls. The critical element is that this return value should show what shows up for resource ownership when the client looks up ownership through methods like VirtualMachine.getProviderOwnerId().
The one set of services that all Dasein Cloud implementation requires is DataCenterServices. MyCloud.java already contains a getDataCenterServices() method that returns an instance of Geography.java. You should therefore implement Geography.java as your first programming task.
This class is generally the easiest and the hardest to implement. It's the easiest because it has the fewest methods you absolutely must edit. It is the hardest because the model mapping is generally not straightforward.
In DataCenterServices, your task is to map the Dasein Cloud concepts of "region" and "data center" to whatever geography exists with the underlying cloud provider. In AWS, it's a simple task because AWS region = Dasein Cloud region and AWS availability zone = Dasein Cloud data center. Very few other clouds have this kind of division.
In most cases, you end up with one or more regions that each map to a single data center. Or you have a single region that maps to multiple data centers. Here's the rule of thumb on how to map concepts:
- A region must fit within a single jurisdiction
- No resources are shared across regions
- Some resources are shared across data centers
- Data centers represent physical proximity
- Block volumes may not be attached to virtual machines in another data center (NFS volumes are not so constrained)
When in doubt, please post to the Dasein Cloud mailing list at https://groups.google.com/forum/?fromgroups=#!forum/dasein-cloud.
All other services are optional, within reason. Typically the process works like this:
- Create a package for the service (like org.dasein.cloud.aws.compute for Compute Services)
- Create a class that extends the abstract version of the service in that package (like AWSCompute extends AbstractComputeServices)
- Implement a constructor that takes your cloud provider as an argument
- Create a getWhateverServices() method in your cloud provider class (the thing that is MyCloud.java in the skeleton). The method, of course, needs to match the signature in CloudProvider since you are overriding a default method that returns null. In the case of compute services, it is getComputeServices()
- For every resource you wish to support, you will then implement a support class
Implementing support classes is where the real work takes place. A services object is simply an enumerator of what resources that services support. For example, compute services can optionally support images, virtual machines, volumes, snapshots, and more. To implement support for virtual machines, you might create a vm sub-package and then create a MyCloudVMSupport class that implements VirtualMachineSupport. You then override the getVirtualMachineSupport() method in your compute services class and implement all of the VirtualMachineSupport methods in your MyCloudVMSupport.java class to do everything necessary to support VM interaction.
Dasein Cloud includes a test suite (already referenced in the skeleton pom.xml) that you can use to verify that your implementation of Dasein Cloud is working the way a client would expect. The way to run the full test suite is the command:
mvn -Pmyprofile clean test
In this scenario, "myprofile" references a profile in your ~/.m2/settings.xml file. Here's a sample for AWS:
<profile>
<id>aws</id>
<properties>
<endpoint>http://ec2.us-east-1.amazonaws.com</endpoint>
<accountNumber>123456789012</accountNumber>
<apiSharedKey>ABCDEFG123456QWE098Q</apiSharedKey>
<apiSecretKey>blahlblahsecretblahblah</apiSecretKey>
<cloudName>AWS</cloudName>
<providerName>AWS</providerName>
<regionId>us-west-1</regionId>
<x509Cert>/Users/me/keys/cert-mykey.pem</x509Cert>
<x509Key>/Users/me/keys/pk-mykey.pem</x509Key>
<test.dataCenter>us-west-1b</test.dataCenter>
<test.region>us-west-1</test.region>
<test.firewall>default</test.firewall>
<test.machineImage>ami-63b8e526</test.machineImage>
<test.product>m1.small</test.product>
<test.shareAccount>135260606263</test.shareAccount>
</properties>
</profile>
Some of the values represent configuration items for a provider context. The rest represent resources to use for testing purposes.
Not all of the entries are necessary. For example, most clouds don't leverage X509 keys (technically, the way AWS uses them, you don't need them for the integration tests). If the cloud doesn't support sharing, you don't need to specify a sharing account.
Note: These tests will create and destroy resources in the cloud. They will cost money to run in a public cloud.
For the most part, the tests do a good job of destroying anything they create and not touching any existing infrastructure. However, in some extreme circumstances (especially failed tests), it is possible that some temporary resources will NOT get destroyed. Make sure you clean up your account after any testing!
The Dasein Cloud team will not push any clouds to Maven Central unless the integration tests all complete successfully. However, it is possible in some cases that the tests are not suited well for your cloud. In those cases, it's better to notify the Dasein team and modify the test than to allow a test to continue to fail. Such cases may also require changes to dasein-cloud-core to enable client discovery of the differences with this cloud.
Finally, you rarely want to run all tests at once. In fact, it takes a long, long time to run all tests. The way to run a single test is to use the command line:
mvn -Pmyprofile -Ddasein.inclusions=SomeTestCase clean test
Where SomeTestCase is the name of a Dasein Cloud test. For example, VirtualMachineTestCase is the test case for testing virtual machine support. See dasein-cloud-test for a full list of test cases.
- Annotate all method return values and parameters with @Nonnull or @Nullable if they are objects, @Nonnegative (if appropriate) if they are raw numerics
- Implement API tracing on all API calls and APITrace.begin()/APITrace.end() on all synchronous operations
- Implement wire logging of all non-SOAP interaction
- Place a header in each file that includes the author of the code, the origin version, and the current version
- If editing an existing class, add your name to the author's list
- Pass all integration tests
- Use the mailing lists and get help when needed!
It's easy to debug code through the integration tests, since they are in essence just the JUnit tests. Individual tests can be run with Maven with -Ddasein.inclusions parameter, and for debugging it's important to instruct Maven not to fork additional process.
Example launch configuration in IntelliJ:
