app

app - A dependency orchestration and application framework

• app - A dependency orchestration and application framework
  • Do You Need This?
  • How To Use
    • Wrapping Code With Factories
      • Factories With Dependencies
    • Defining Dependencies
      • Kinds Of Dependencies
    • Defining And Linking Implementations
    • Installing Runtime Hooks
    • Combining Everything Into An Application
    • Applying Configuration Values
  • Extending The Project
    • Building A Selector
    • Building A Loader
  • Best Practices
    • Loose Coupling
    • Use A Dedicated Packages
    • Add Validation To Your Build
      • Validating Factories
      • Validating Implementations And Hooks
      • Validating Applications
  • Planned Features
  • Developing
    • Make targets
  • License

Do You Need This?

This project is designed to help system, or service, developers to create and manage a runtime. Specifically, this projects provides for:

• Dependency orchestration and injection
• Runtime hooks for custom start and stop behaviors
• Integration with configuration systems
• Signal handling

If you are building a library then you do not need these features as they are specific to managing runtimes.

How To Use

Wrapping Code With Factories

This project is designed to be applied as a set of layers on top of otherwise functioning Go code. We value loose coupling and have built this framework such that no amount of this framework should be present in core application code. To that end we have developed a structure we call the Factory Protocol that can be applied around any existing code that constructs a value you want to have injected as a dependency.

The Factory Protocol is an abstract interface that would normally be defined using generics or templates in other languages. However, Go generics are still some time away from inclusion in the language so we cannot define the protocol using an actual type definition. Instead we will use pseudo-code:

type Protocol interface{
    Config(ctx context.Context) *C
    Make(ctx context.Context, conf *C) (T, error)
}

In this definition C refers to a struct you will define that acts as a data container for all configuration data needed to create the instance. The Config method must return a pointer to an instance of C that contains any default configuration values.

The instance of C returned from Config will be processed using a configuration loading system. By default we use stackopsd/config but the section titled Extending The Project details how other configuration systems may be used. Once the configuration processing is complete the Make method is called with the instance of C to obtain an instance of T. The type T is any type your code constructs whether that is a primitive value, such as a string, or an instance of a struct, pointer to a struct, or an interface type, etc.

The names an definitions of C and T are entirely up to you. For example, if we wanted to create a factory that generates *http.Client instances then we might do the following:

type Configuration struct{
    Timeout time.Duration
}

type Factory struct {}

func(*Factory) Config(ctx context.Context) *Configuration {
    return &Configuration{Timeout: 5*time.Second}
}

func(*Factory) Make(ctx context.Context, conf *Configuration) (*http.Client, error) {
    return &http.Client{Timeout: conf.Timeout, Transport: http.DefaultTransport}, nil
}

This example defines a configuration struct called Configuration that contains an input value needed by the Make method. The Make method constructs an instance of *http.Client. Having a configuration struct and a constructor method is a fairly common practice in Go. This is merely a structure that encapsulates those behaviors in a way that can be integrated with configuration loading system.

Factories With Dependencies

If your factory requires things beyond configuration values then those dependencies should be defined as attributes of the factory struct. To illustrate we will modify the previous factory to depend on an http.RoundTripper rather than using the Go default:

type Configuration struct{
    Timeout time.Duration
}

type Factory struct {
	Transport http.RoundTripper
}

func(*Factory) Config(ctx context.Context) *Configuration {
    return &Configuration{Timeout: 5*time.Second}
}

func(f *Factory) Make(ctx context.Context, conf *Configuration) (*http.Client, error) {
    return &http.Client{Timeout: conf.Timeout, Transport: f.Transport}, nil
}

The factory is nearly identical except that it now references a Transport value that is attached to the factory. This is the recommended way to manage dependencies for a factory for easiest integration with the system later.

Determine the difference between a dependency and configuration is an important step. The rule of thumb is that any complex object, like our http.RoundTripper in the example, is a dependency that will be constructed separately to our factory and any value we expect a human to set is configuration that should be in our struct.

For more on the Factory Protocol see https://github.com/stackopsd/factory#the-factory-protocol.

Defining Dependencies

After defining a set of factories that wrap all of your dependencies it's time to then actually declare the dependencies for the system. To do this we recommend defining a function like:

func CreateDependencies(ds app.Dependencies) (app.Dependencies, error) {
	httpClient := app.NewDependency("http_client")
	
	return append(ds, httpClient)
}

Each dependency required by an application must be defined first as an abstract dependency. The name given to the dependency must be unique across all dependencies for the system.

Kinds Of Dependencies

The default kind of dependency returned from NewDependency is a "driver". A driver dependency may have one or more implementations registered but only one will be selected at runtime. This is the most common form of dependency. There is also a second kind of dependency, returned from NewList, that can tolerate having zero or more implementations active simultaneously. This kind of dependency is less common but can be used to support dynamic middleware and optional decorator chains.

Defining And Linking Implementations

With some set of abstract dependencies defined you then define some number of concrete implementations for those dependencies and link them:

httpClient := app.NewDependency("http_client")
httpTransport := app.NewDependency("http_transport")

httpClientImpl, err := app.NewImplementation(
	"default",
	func(ctx context.Context, rt http.RoundTripper) (*Factory, error) {
		return &Factory{Transport: rt}, nil
	},
)
if err != nil {
	return nil, err
}
httpClientImpl.Implements(httpClient)
httpClientImpl.Requires(httpTransport)

Each implementation must be given a name that is unique within the dependencies to which it is linked. Each implementation must also provide a constructor function that accepts any dependencies, shown as attributes of the factory struct in the previous examples, and returns an instance of the factory that can create the implementation. The process of called NewDependency will validate the constructor function so it is important to handle any errors returned.

Once the implementation is created successfully you then must link it to both any dependencies it implements and any dependencies it requires. This is done using the Implementation.Implements and Implementation.Requires methods. Calls to Requires take any number of dependencies in the order they are requested by the constructor function. Each call to Requires overwrites any previous settings with a new set of dependencies.

Note: If a requirement is a "list" kind of dependency then the parameter of the constructor function must be a slice.

Installing Runtime Hooks

Once you have all of your dependencies and implementations linked then you may need to define some runtime hooks. Hooks are processed at system start up or shut down and are the appropriate place to perform actions such as starting goroutines for network listeners, etc. Defining hooks works very much like defining implementations. Each hook is wrapped in a factory and each hook must be linked to the various dependencies it requires:

type ServerStarterConfig struct {}
type ServerStarterFactory struct {
	Server *http.Server
}
func (*ServerStarterFactory) Config(context.Context) *ServerStarterConfig {
	return &ServerStarterConfig{}
}
func (f *ServerStarterFactory) Make(context.Context, *ServerStarterConfig) (<-chan error, error) {
	ch := make(chan error)
	go func() {
		err := f.Server.ListenAndServe()
		if err == ErrServerClosed {
			ch <- nil
			return
		}
		ch <- err
	} ()
	return ch, nil
}

server := app.NewDependency("server") // assume this will be an http server
startServer, err := NewHook(
	"start_server",
	func(ctx context.Context, srv *http.Server) (*ServerStarterFactory, error){
		return &ServerStarterFactory{Server: srv}, nil
	},
)
if err != nil {
	return nil, err
}
startServer.Requires(server)

By making hooks into factories we gain the ability to provide for both dependency injection and configuration in the same way we expect for implementations. One notable difference is that all hook factories must return a type of <-chan error. This channel will be treated as a signal for a critical system failure. In the above example the channel receives a non-nil error value if the server stops unexpectedly. The result will be termination of the application. Alternatively, if the server stops as expected then we write a nil value to the channel instead. This pattern enables for hooks that start long-running goroutines to report if those goroutines crash in a way from which the system cannot recover.

Hook registration is done with the OptionStartHook and OptionStopHook modifiers when creating an Application container.

The NewHook method validates the signature of the hook function so it is important to handle any errors returned.

Combining Everything Into An Application

Finally, all of the defined dependencies, implementations, and hooks are brought together in an Application instance:

package main

import (
	"github.com/stackopsd/app"

	"github.com/myuser/myproject/pkg/runtime" // the package where you defined everything
)

func main() {
	ctx, cancel := context.WithCancel(context.Background())
	defer cancel()

	deps, err := runtime.CreateDeps()
	if err != nil {
		panic(err)
	}
	startHooks, err := runtime.CreateStartHooks(deps)
	if err != nil {
		panic(err)
	}
	stopHooks, err := runtime.CreateStopHooks(deps)
	if err != nil {
		panic(err)
	}
	a := app.NewApplication(
		ctx,
		app.OptionDependencies(deps...),
		app.OptionStartHooks(startHooks...),
		app.OptionStopHooks(stopHooks...),
	)

	if err = a.Start(ctx); err != nil {
		panic(err)
	}
	if err = <-a.Wait(); err != nil {
		panic(err)
	}
	if err = a.Stop(ctx); err != nil {
		panic(err)
	}
}

Applying Configuration Values

Because it is a part of our ecosystem, we ship this project with built-in support for using https://github.com/stackopsd/config for managing the loading of configuration values. The default application will use environment variables as the source of configuration. If you want to use config but with a JSON or YAML file instead then you can swap out the defaults using the OptionConfigLoader feature:

confLoader, err := config.NewLoaderFile("config.json_or_yaml")
if err != nil {
    panic(err)
}

app, err := depo.NewApplication(
    context.Background(),
    depo.OptionConfigLoader(confLoader),
)
if err != nil {
    panic(err)
}

By default, the system will attempt to load configuration in order to select which implementations are active at any given time as well as loading relevant configuration values for the implementation factories. The configuration pattern to use for selection of implementations is:

APP__DEP_ONE__DRIVER="implementation_name"
APP__DEP_ONE__IMPLEMENTATION_NAME__CONFIG_VALUE_ONE="something"
APP__DEP_ONE__IMPLEMENTATION_NAME__CONFIG_VALUE_TWO="something2"
APP__LIST_ONE__ENABLED__0="implementation_one"
APP__LIST_ONE__ENABLED__1="implementation_two"
APP__LIST_ONE__ENABLED__2="implementation_three"
APP__LIST_ONE__IMPLEMENTATION_ONE__CONFIG_VALUE_ONE="something"
APP__LIST_ONE__IMPLEMENTATION_ONE__CONFIG_VALUE_TWO="something2"
APP__LIST_ONE__IMPLEMENTATION_TWO__CONFIG_VALUE_ONE="something"
APP__LIST_ONE__IMPLEMENTATION_TWO__CONFIG_VALUE_TWO="something2"
APP__LIST_ONE__IMPLEMENTATION_THREE__CONFIG_VALUE_ONE="something"
APP__LIST_ONE__IMPLEMENTATION_THREE__CONFIG_VALUE_TWO="something2"
APP__ON_START__HOOK_ONE__CONFIG_VALUE_ONE="something"
APP__ON_STOP__HOOK_ONE__CONFIG_VALUE_ONE="something"

or the same configuration express as YAML:

dep_one:
	driver: "implementation_name"
	implementation_name:
		config_value_one: "something"
		config_value_two: "something2"
list_one:
	enabled:
		- "implementation_one"
		- "implementation_two"
		- "implementation_three"
	implementation_one:
		config_value_one: "something"
		config_value_two: "something2"
	implementation_two:
		config_value_one: "something"
		config_value_two: "something2"
	implementation_three:
		config_value_one: "something"
		config_value_two: "something2"
on_start:
	hook_one:
		config_value_one: "something"
on_stop:
	hook_one:
		config_value_one: "something"

The important notes are that each dependency must define an active driver. Each list is disabled by default and each value must be enabled in a list. All configuration for a dependency is rooted under a key that matches the dependency name. All configuration values of the factory structs are rendered according to the process documented in https://github.com/stackopsd/config.

Generating Sample Configurations

This project comes bundled with methods called HelpENV and HelpYAML that generate sample configuration files for an application using the structure that will be read by the https://github.com/stackopsd/config loader that is also bundled.

Extending The Project

In addition to having a low impact on application code, we also want this project to be extensible to specific project needs. We do bundle in support for http://github.com/stackopsd/config because that's our configuration loading system but we never want that to be the only configuration system possible. Likewise we make some opinionated decisions about how driver and list implementations are selected that may not be correct in all cases. For example, what if you wanted to have list items enabled by default?

To enable modifications of all these behaviors we provide three interfaces that need to be implemented: The Selector, the Loader, and the HooksExecutor.

Building A Selector

A Selector is a component used by the system to filter out any registered implementations that will not be loaded when the system starts. The Selector interface is:

// Selector filters the implementations of each Dependency to only those that
// will be loaded at runtime. Any selection process may be used so long as the
// DepdendencyKind rules are maintained such that each DependencyKindDriver has
// exactly one implementation, and each DependencyKindList has zero or more
// implementations. Whatever remains in the implementations lists after this
// component processes them will be loaded.
type Selector interface {
	Select(ctx context.Context, ds []depo.Dependency) ([]depo.Dependency, error)
}

Like the code documentation says, any method may be used to filter the implementations so long as the rules for each kind of dependency are preserved. This is the component to create if you want to change how implementations are selected. For example, this is where you would implement lists that default to "enable all" or drivers that automatically select the first implementation if there is only one.See the LoaderConfig implementation in this project for an example of how filtering may be done.

Building A Loader

A Loader is the component that actually constructs an instance of the factory output for all implementations. A Loader must implement this interface:

// Loader implementations construct instances of all the implementations
// requested.
type Loader interface {
	Load(ctx context.Context, ds []depo.Dependency) ([]depo.LoadedDependency, error)
}

The dependency set given to the Loader will already be filtered to selected implementations and ordered such that each implementation is guaranteed to come after anything is depends on. This is the component to create if you want to change how factory configuration structs are populated. See the LoaderConfig for examples.

Building A HooksExecutor

A HooksExecutor is the component that constructs each hook and loads its configuration. A HooksExecutor must implement this interface:

// HooksExecutor implementations are responsible for executing a set of hooks.
type HooksExecutor interface {
	ExecuteHooks(ctx context.Context, hooks []Hook, loaded []depo.LoadedDependency) (<-chan error, error)
}

The loaded value will contain all loaded dependencies and hooks will contain all hooks for a given lifecycle event. See the HooksExecutorConfig implementation in this project for a detailed example.

The application accepts a HooksExecutor for each event through the OptionStartExecutor and OptionStopExecutor modifiers when creating an Application.

Best Practices

These are our current recommendations for how to most effectively use this, and any, dependency orchestration and injection system. They are likely to evolve over time so check back periodically.

Loose Coupling

This library should be used as a layer on top of otherwise valid Go code. This library should never be imported or referenced outside of code that is setting up a runtime, such as main.go. Your base code, your system design, and even your factory implementations should be free of any imports or references to this library.

Use A Dedicated Packages

We recommend using a dedicated package, such as /internal, /pkg/internal, or /pkg/runtime, etc., to contain the dependency orchestration code. This will both help prevent tight coupling inside application code and provide a clear place where new system dependencies are registered and organized. We recommend also breaking up the orchestration into multiple steps that can be leveraged by 3rd parties and tests. To illustrate, we recommend your runtime package have the following exports:

package runtime // or other name as desired

// Dependencies is a container for all of the dependencies the system will
// need. Having this both helps create a singular location where all system
// dependencies are defined and provides structured access to dependency objects
// though field names.
type Dependencies struct {
	Example depo.Dependency
	Example2 depo.Dependency
}

// NewDependencies creates a populated instance of your container.
func NewDependencies() (Dependencies, error) {
	return Dependencies{
		Example: app.NewDependency("example")
		Example2: app.NewDependency("example2")
	}
}

// RegisterImplementations is where you attach all normal implementations to
// the various dependnecies.
func RegisterImplementations(ds Dependencies) error {
	implementation, err := app.NewImplementation("implementation", ...)
	if err != nil {
		return err
	}
	implementation.Implements(ds.Example)
	implementation.Requires(ds.Example2)

	return nil
}

// CreateStartHooks centralizes the process of adding startup behavior.
func CreateStartHooks(ds Dependencies) ([]app.Hook, error) {
	hook, err := app.NewHook("hook", ...)
	if err != nil {
		return err
	}
	hook.Requires(ds.Example)

	return []app.Hook{hook}, nil
}

// CreateStopHooks centralizes the process of adding shutdown behavior.
func CreateStopHooks(ds Dependencies) ([]app.Hook, error) {
	hook, err := app.NewHook("hook", ...)
	if err != nil {
		return err
	}
	hook.Requires(ds.Example)

	return []app.Hook{hook}, nil
}

// DependenciesToList is an explicit mapping of all the values in the struct
// to a slice. This helps enable 3rd parties to append to your set of
// dependencies before starting an application.
func DependenciesToList(ds Dependencies) []depo.Dependency {
	return []depo.Dependency{
		ds.Example, ds.Example2,
	}
}

// NewApplication brings together all the other methods to generate the final
// output.
func NewApplication() (app.Application, error) {
	ds := NewDependencies()
	if err := RegisterImplementations(ds); err != nil {
		return nil, err
	}
	startUp, err := CreateStartHooks(ds)
	if err != nil {
		return nil, err
	}
	shutDown, err := CreateStopHooks(ds)
	if err != nil {
		return nil, err
	}
	return app.NewApplication(
		context.Background(),
		app.OptionDependencies(DependenciesToList(ds)...),
		app.OptionOnStart(startUp...),
		app.OptionOnStop(shutDown...),
		// Install any other options here as needed.
	)
}

Add Validation To Your Build

This project uses reflection to perform most of its complex functions which means we've had to sacrifice a large amount of compile time protection. To account for this we also provide a fairly extensive suite of runtime protections by recreating many of the checks that the compiler would have performed for us. However, because our tooling operates at runtime it means you must have something that executes the validation code in order to report on pass or fail. We recommend running validation as a unit test.

Validating Factories

If you don't mind pulling in a Go project for testing then we recommend using https://github.com/stackopsd/factory to validate your factory implementations:

func TestMyFactory(t *testing.T) {
	if err := factory.Verify(&MyFactory{}); err != nil {
		t.Error(err)
	}
}

The Verify tool from our factory library will report on whether or not the given instance satisfies the Factory Protocol and give details on any aspect of the instance that does not.

Validating Implementations And Hooks

Implementation constructors and hook functions are validated during their respective calls to NewImplementation and NewHook. If you are using our suggested practice of using methods to create both of these resources then you can add a test like:

func TestImplementationsAndHooks(t *testing.T) {
	deps := NewDependencies()
	if _, err := CreateStartHooks(deps); err != nil {
		t.Fatal(err)
	}
	if _, err := CreateStopHooks(deps); err != nil {
		t.Fatal(err)
	}
}

Validating Applications

Finally, there is some validation that can only be done when all the pieces are available together. Notably, validation of dependency cycles, missing dependencies, and advanced type protection can only happen when all values are together. To help with this we provide a method on the Application called Validate that performs all runtime validation:

func TestApplication(t *testing.T) {
	app, err := NewApplication()
	if err != nil {
		t.Fatal(err)
	}
	if err := app.Validate(); err != nil {
		t.Fatal(err)
	}
}

Planned Features

This project is still very young and there are a few features we'd like to deliver in the future:

• Dependency graph generation.

  Especially for large, complex projects and when debugging cycles, it would be useful to visualize the dependency graph of a system. Because this feature would likely bring in a number of unrelated dependencies we may implement this as a separate module.

• Code generation.

  Even though we've re-implemented most of the compiler type checks, it may still be a useful feature to have an option of generating the final orchestration code.

Developing

Make targets

This project includes a Makefile that makes it easier to develop on the project. The following are the included targets:

• fmt

  Format all Go code using goimports.

• generate

  Regenerate any generated code. See gen.go for code generation commands.

• lint

  Run golangci-lint using the included linter configurations.

• test

  Run all unit tests for the project and generate a coverage report.

• integration

  Run all integration tests for the project and generate a coverage report.

• coverage

  Generate a combined coverage report from the test and integration target outputs.

• update

  Update all project dependencies to their latest versions.

• tools

  Generate a local bin/ directory that contains all of the tools used to lint, test, and format, etc.

• updatetools

  Update all project ools to their latest versions.

• vendor

  Generate a vendor directory.

• clean/cleancoverage/cleantools/cleanvendor

  Remove files created by the Makefile. This does not affect any code changes you may have made.

License

Copyright 2019 Kevin Conway

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.