Code-Test-Document ( ver 1.1 )

A reasonable way of Code-Test-Document triad organization in a ruby based application with a conceptions and tools overview and comparision, containing some code examples.

Back then

When I was doing my first steps as C++ Win32 programmer, I was surprised how much Microsoft valued it’s documentation. A MSDN documentation package was a couple of times bigger than the MS Visual Studio, and tech writers staff at Microsoft, as I heard, was at least as big as programmers or may be even bigger!

At that time seeing automated testing in UI application was like seeing unicorn at the bus station. All the testing were done manually by the QA department.

Now

Now time has changed. Today programmers write their own tests, and, sometimes :), even before writing meaningful code. But what about Documentation? Does programmers need to write it?

If programmer delivers something solid and ready for the end user, like an old-school rails application, he may not. But in case its facing other programmer or automations, like when you deliver an API-service from a server backend to a mobile front, you better start writing some memo on how to use it.

There is no such thing as good undocumented API.

Selecting specification

I will not reinvent bicycle, I’ll just quote:

Swagger

• Pros: Heavily adopted, large community of users and supporters, greater support for multiple languages
• Cons: Lacks advanced constructs for metadata

RAML

• Pros: Supports advanced constructs, decent adoption, human readable format, high industry backing
• Cons: Lacks code-level tooling, still unproven long-term

API Blueprint

• Pros: Easy to understand, simple to write
• Cons: Low adoption, lacks advanced constructs in general, complex installation

JSON:API (v 1.1 https://jsonapi.org)

• Pros: has fast serilializer
• Cons: Low adoption

Need more details follow the link: Top Specification Formats for REST APIs

I did some digging myself and also chose the swagger.

Swagger is the world’s largest framework of API developer tools for the OpenAPI Specification(OAS)

It goes with rich opensource and commercial sets of tools, a large comminity and a multiple languages code base. When I started latest search on the best way to document API, I didn’t found something nearly comparable to swagger. ( Rem. at the moment when the first version of this read was released 2017 )

Here’s swagger to your stack

Choose right

The best way to deal with Code-Test-Document trio is to:

test a code in the way it documented

Then you can be sure: a functionality is working as documented, the key is not only “is working”, but “is working as documented”. So this brings up a first criterion: documentation must be testable.

Keep in mind that you can’t always bring a 100% test coverage for a code, but you must and can do a 100% test coverage for the documentation!

The second criterion is maintainability. If solution can be tested but can’t be properly maintained, it’s practically as bad as untested. So all parts of a stack and solution as whole must be maintainable.

And the last, pretty obvious one, is technological advance. Better to use standard and widespread technology/format/specification than uncommon and custom suited. So solution must be technologically validated, in our terms it means the swagger compatibility of solution.

Documentation inside test

The most widespread way of organizing API-documentations is to put documentation inside tests. Yeah! Everybody likes testing, and so they would love to do a more job inside their tests, by adding a documentation code there. Also this will bring more readability and maintainability to your tests and documentation (hint: nope it will not)

It’s like mixing a salt and a peper trying to get a sugar, it doesn’t work like that. To be honest it was a first idea which I strongly considered. One of a great pros was a strong set of tools and examples. RSpec-based tools like: Rswag, former swagger_rails or rspec_api_documentation with example: How we made writing test fun and easy; MiniTest-based tools and examples: Minitest-apidoc, mininitest-apidoc example.

What’s wrong with this solutions and ideas? I think they are not flexible and its hard to maintain them.

First of all RSpec solutions bring yet another DSL to your application.

Second, lets assume you want to do an integration testing, something more complicated than GET /resource/id. This is the time it will become a real mess, either you are gonna “document” every piece of it, welcome mess, or you will use different sets of test for it, one simpler is documented and the second complex is not, which sooner or later will break away from documented cases.

Code examples

RSpec api documentation:

Final DSL example from “How we made writing tests nice and easy”

Honestly, I dont know how much time do I need to get used to this type of document+test mix, and how much new teammate would spend on accepting this type of code/test/document organization.

Rswag:

rswag example from rswag repo

Simple example — simple code. It’s readable, but still test and doc are mixed, and only one step of complexity will turn a mix to a mess.

mini-test apidoc:

minitest-apidoc

The only plus for this tool and a code snapshot — is MiniTest :). Any complexity in data, test-cases and so will bloat it up to enormous sizes, breaking up any maintainability.

Score:

Maintaiability — not much, RSpec custom DSLs, blending test with a documentation, makes it hard to read, understand and maintain. Indie minitest apidoc is too raw, and suffers the same problem except for DSL.

Testability — average, generated documentation is tested, but looks like any complex integration tests are not for this kind of solution.

Swaggerability — some tools are compatible, some are not. Plus for rswag, strong minus for minitest-apidoc and rspec_api_documentation.

Documentation inside code

Second widely advocated idea is to document API inside API-code, i.e. place documentation in the controller. You can get some examples of it in: Do the Right Thing and Document Your Rails API with Swagger and Apipie — amazing tool for documenting your Rails API.

First of all, this may not be considered as complete solutions, cause testing part usually stays out of the discussions scope. Second reason against placing documentation inside a code, is a new shades of mess, previously there was a disputable mix of docs and tests, now we bring indisputable mess to the controller.

Of course we can move it to a module, but this is nothing different from the last one way of organization, where documentaion is separated from code and test, so for now I’ll keep it the way it was intend to be in the first place.

Code examples:

apipie:

Look for “Some application code” lines,

The code to documentation ratio is 3 to 1, now you can start your quest for searching for application code.

swagger-docs:

swagger-docs example

Same as above, you need to look hard to separate the documentation meta-code from your application code.

Score:

Maintainability — not much, blending controllers code with multiple times more documentation meta-programming code breaks bad the ‘skinny controller’ idea, and also messing up the application code with the documentation.

Testability — may be high with a help of external tools, cause usually this idea serves only documentation pupose, so a result can be effectively tested with appropriate tools.

Swaggerability — again plus for swagger-docs, minus for apipie.

Standalone documentation with automated tests against it

You can get the idea from Automating Empathy: Test Your Documentation With Swagger and Apivore. This is a great example of how things must be!

In short: you write a documentation the best way suitable for you, but apart from a code and tests. It can be done even in a raw json or a special editor, you are free to do this with a max comfort. When the documentation is done it can be used by a test tool which will check a response structure; routes, code and documentation coverage.

Code examples:

swagger-blocks ( documentation example):

swagger-blocks example, but you can choose other tool for documenting

Remember it is not mandatory to use a swagger-blocks gem, if you don’t like this tool — choose different. You are OK, as long as chosen tool swagger-compatible and testing tool supports its swagger-revision.

mini-apivore ( test example ):

This is a complete example on RESTful resource named Card.

With a proper set of helpers, wrappers of the original root check_route method, you can get any readability level you want. It’s like using plain MiniTest helpers, but with power of documentation based testing ( documentation coverage, response structure check, action and response coverage, etc..)!

Score:

Maintainablity — high and flexible. For documentation and tests you can choose different tools, with a proper tools you can even get technical writers to do some part of a job.

Testablity — high, but only with proper tools. For a swagger based documentation and a ruby stack, Apivore and MiniApivore works fine.

Swaggerability — high, depends only on the testing framework. Current ruby testing tools for swagger supports Open API specification version 2.0 and lower. ( Rem. at 2017 )

With this approach you can document as you like, and test in the way you already do.

Resume

Right conception:

1. Separate the code, the test and the documentation from each other
2. Test a code in the way it documented.
3. Use a swagger as it brings the best technology pool

The stack I use to implement this conception:

1. The rails as a code base.
2. The MiniTest as test framework, and MiniApivore as swagger schema test helper.
3. The Swagger-blocks as a schema builder, but strongly customized. I’ve added DSL for schema inheritance for models description, RESTful routes chortcuts, array shortcuts, etc.
4. The swagger_ui_engine as documentation viewer.

References:

API specifications:

1. Top REST-API specifications format
2. Another opinion on swagger, RAML, API blueprint
3. RESTful API description languages wiki
4. API-Blueprint VS Swagger ( swagger wins by KO )
5. Swagger

Swagger-tools:

1. Swagger ruby OS tools ( all of below links are presented there)
2. Swagger-blocks ( documentation )
3. Rswag, former swagger_rails (rspec injected documentation)
4. MiniApivore for mini-test, Apivore for RSpec

Documentation tools unrelated to swagger:

1. Minitest-apidoc, document inside minitests tests. Example on minitest-apidoc
2. API-blueprint. Dox exmaple for API-blueprint
3. Apipie for rails
4. rspec_api_documentation ( RSpec based documentation tool with multi-format exports )

Examples and conceptions:

1. DDD — documentation driven development manifesto
2. Do the Right Thing and Document Your Rails API with Swagger, swagger-docs example on documenting rails API. Two weak points: first — how to deal with test does not described in this note, and second — it brings mess to controllers. Placing doc inside a code is wrong idea.
3. Automating Empathy: Test Your Documentation With Swagger and Apivore, Swagger + Apivore as document driven development. The very best long read, which gave me the right idea on how the things are need to be done.
4. Producing documentation for your rails api. Codeship nice, clear and usefull as usual. Describes tools, practicies and conceptions around documentation and testing, with links and examples.
5. How we made writing test fun and easy. Rspec way to test and document. As far as I dislike RSpec, this is not the way for me either. Also I don’t like the idea of mixing test and doc in one file.
6. Building an API with Swagger JSON API and Rails (Part 2) Some examples on swagger API description, didn’t get much from this.