How to implement an architecture workbench and ArchGuard ?

Building an architecture workbench is not an easy task, involving a series of compiler-related knowledge, editor-related knowledge, and of course, its core architecture-related knowledge.

We eventually launched an alpha version of the first “useful” Architecture Workbench (how do you define usable?) at the end of May, following a succession of glitches and bumps. In this early demo version, you can try the architecture-as-code philosophy we built in ArchGuard, and how to build a workbench around a system? What’s more, after you’ve mastered the capacity to create workbenches, you’ll see them everywhere, even API workbenches.

We explained the concepts underlying the Architecture Workbench, as well as its basic principles and practices, in a previous ” Architecture Workbench ” linked post. In this article, we will continue to introduce:

• How ArchGuard implements such a workbench.
• We went through a number of architectural considerations when constructing this PoC (Proof of Concept). Why A and not B
• Some short code samples.
• issues that arose during the procedure

Of course, to obtain more extensive information, you must install the most recent code from GitHub. You can also test the Demo by using curl -s https://archguard.org/install.sh. | bash -s master(This script does not seem to support Windows). If you’ve previously created the matching environment locally, all you need to do is run docker-compose pull && docker-compose up:

Prototype Reference and Design: Interactive Environment and Documentation Experience

What is a document? What is code? There is no clear boundary between the two, the document is executable, and the code is also executable. In their final form, however, they are all knowledge. So, the focus remains on how to make this knowledge explicit. So from the prototype reference, we focus on interactive environment and document experience design.

Interactive Environment: Jupyter & Zeppelin & Nteract

As the industry representative of interactive programming, Jupyter became the first object of our research. However, in terms of implementation, we don’t learn much (“code”) from its own source code. On the contrary, around its ecology and competitors, we have seen some more interesting highlights, such as Kotlin Jupyter, Zeppelin, Nteract, etc.

• Nteract provides a series of components, SDK to build interactive applications, such as messaging and so on. However, Nteract was designed mainly for use in the Electron environment, so some libraries cannot be used, such as ZeroMQ – designed only for the Node environment.
• Zeppelin builds a simpler execution environment (Interpreter) that provides some more interesting implementation-level abstractions than Jupyter’s Kernel API.
• Kotlin Jupyter became a cornerstone of our current implementation. Because it is still in the early experimental stage, we have encountered a series of missing dependencies during the build process.

Looking back, we should need to go back and look at Jupyter’s abstract interface, perhaps to provide more ideas.

Document reading experience and document engineering experience

For the documentation experience, I’ve always advocated focusing on two parts:

• Document reading experience. That is, the document experience provided to the reader.
• Document engineering experience. That is, the documentation writing experience provided to the project. (This part is often overlooked)

For the document experience, in addition to various architecture diagrams for declarative programming, various definition capabilities need to be provided. The main sources of reference are the documentation of programming languages ​​in our daily development.

And diagrams as code, such as Mermaid and Graphviz, provide a good balance between the two (for programmers only).

Technology Assessment: DSLs, REPLs, and Editors

Coming back to the implementation, in conducting the technical evaluation of the Architecture Workbench, we focused on the DSL (Domain Specific Language) syntax written by the architect, the REPL (read–eval–print loop) runtime environment, and the editor for interaction. Its core focus is: how to build a better developer experience, an old and difficult topic.

DSL syntax: Antlr vs Kotlin DSL

In ArchGuard, the Antlr framework is mainly used to parse different languages ​​(ie Chapi). As a result, there is no technical barrier to using Antlr to create a new DSL and associated compiler front-end. Even, in the past experience, we also have the experience of large-scale IDEA plug-in architecture design and development.

However, for DSL, the core factors we have to consider are:

• The cost of learning grammar.
• Grammar for experience design.
• Syntax editor/IDE support.

If grammar is just an API of a language, it can greatly reduce the cost of learning. Although Kotlin is a bit unfamiliar, Groovy + Gradle is very familiar. Therefore, the way we use is to build the construction DSL based on the Type-safe builders that come with the Kotlin language. Ktor’s routing example is an official reference example:

routing {

    get("/hello") {

        call.respondText("Hello")

    }

}

In addition to the already rich IDE, editor support. When constructing the architectural fitness function, the mathematical functions provided by the language library can also be used to customize various calculation rules.

Architecture REPL: Kotlin Scripting vs Kotlin Jupyter

For building an interactive architecture REPL, we need to consider a core point: building an execution context (EvalContext). That is, the code to run later depends on the context provided by the previous code, such as variables, etc.: val x = 2 * 3, which can be used later x.

For us, there are two options:

• An experimental feature built into the Kotlin language: Kotlin Scripting provides a technique for executing Kotlin code as scripts without prior compilation or packaging into executable files. Because, for us, we just need to build our DSL package and we can execute it directly.
• The implementation of Kotlin Jupyter also provides a series of API encapsulations based on Kotlin Scripting.

On the REPL, at first, we struggled with implementing it ourselves, or based on Kotlin Jupyter, after all, Jupyter contains a bunch of unnecessary code. Later, I found that the code is very complicated. Although it is all MIT protocol, we do not want to maintain a downstream version of unstable functions.

So, in the end, we built ArchGuard’s architectural REPL based on Kotlin Jupyter’s API.

Editor of Discovery: ProseMirror vs. Others

For the editor, the core point to consider is: component extensibility. That is, components for displaying charts, or other related components for displaying results can be added as needed.

Jupyter and Zeppelin employ a block (Cell) editor for design, which means that the document is divided into blocks. The slight difference is that Jupyter is based on CodeMirror, while Zeppelin is based on Monaco Editor. This block-based editing function is a bit fragmented, and the interactive experience provided is not friendly to pure keyboard operation.

Therefore, in order to explore a better way of document interaction, we have successively referred to a series of editors: CodeMirror, Draft.js, Lexical, ProseMirror, etc. ProseMirror is another work by the authors of CodeMirror, which combines Markdown with a traditional WYSIWYG editor. That is to say: you can write Markdown or use rich text (PS: When writing this article, the bottom layer of Quake I used is also ProseMirror). That is, it can meet the needs of two types of people, with and without Markdown, they can both get their own mouse (markdown) and keyboard (rich text) from the editor.

After exploring, we found that the rich-markdown-editor based on ProseMirror can provide the required functionality. Only need to write some plug-ins like ProseMirror, no need to write a lot of markdown-related processing functions.

Landing: building data communication and result presentation

In order to verify that the entire PoC (Proof of Concept) is feasible, the next step is to use the data as the glue to connect everything together to build a complete end-to-end example like this:

• Frontend → REPL. Write the DSL on the front end, execute the run, and send the data to the REPL.
• REPL → Frontend. The REPL parses the data and returns subsequent Actions to the front end.
• Frontend → Backend. The front-end decides whether to display the architecture diagram or send a request to the back-end according to the Action.
• Backend → Frontend. The backend executes the corresponding command according to the request of the frontend and then returns the result to the frontend.
• front end. The front-end is then processed according to the data of the back-end.

So, in fact, there is only one core part: the design of the model, such as Message and Action.

The Message Model for Data Transmission and Processing

In the REPL service, after receiving the front-end data through WebSocket, it needs to be converted into corresponding data and returned to the front-end. The following is the Message we defined in the PoC:

data class Message(

    var id: Int = -1,

    var resultValue: String,

    var className: String = "",

    var msgType: MessageType = MessageType.NONE,

    var content: MessageContent? = null,

    var action: ReactiveAction? = null,

)

After executing the code passed in from the front end, some subsequent Action information (in the code ReactiveAction) and corresponding data ( actionin ) will be returned according to different execution results.

REPL: Build Execution Environment

For REPL, we still need to do:

Build the REPL environment. Such as adding the jar package of ArchGuard DSL, and the corresponding Jar of Kotlin Scripting and Kotlin Jupyter.

Add % archguardMagic. Add a custom one LibraryResolver.

Although I am not familiar with REPL, fortunately with the source code of Kotlin Jupyter as a reference, this process is not too painful. Although the process is also extremely painful: no documentation is available, the environment is only designed for Jupyter, and only test cases can be seen. However, at least you can still look at the test cases – tests are a good thing.

In the development environment, the classpath of the Java runtime environment will be loaded (for details, see: KotlinReplWrapper ):

val property = System.getProperty("java.class.path")

var embeddedClasspath: MutableList<File> = property.split(File.pathSeparator).map(::File).toMutableList()

In the runtime environment, only the required jar packages will be referenced. The inconsistency between the two environments also needs to be explored in how to optimize in the follow-up.

editor:

In the process of our landing, the implementation of the editor is divided into two parts, one is to write the ProseMirror plug-in, and the other is to improve the perception of Monaco Editor.

ProseMirror plugin writing

For code blocks, a LivingCodeFenceExtensionplugin replaces the code block syntax function in rich-markdown-editor and point to the Monaco Editor component:

<CellEditor

  language={language}

  code={value}

  removeSelf={this.deleteSelf(props)}

  codeChange={this.handleCodeChange}

  context={this.options.context}

  languageChange={this.handleLanguageChange}

/>

Then around the two editors, a series of interactions are built, such as language change, deletion of code blocks, execution of code, etc.

Building a DSL developer experience around the Monaco Editor

The improvement of Monaco Editor mainly revolves around: adding code highlighting, auto-filling and IntelliSense. Now, only the basic functions have been completed, and there are still many functions that need to be explored later.

Graphics and presentation of results

For results, the core part is ResultDispatcheron, as the name implies, different display results are displayed according to different results, such as:

switch (result.action.actionType) {

  case ActionType.CREATE_REPO:

    return <BackendActionView data={data} actionType={BackendActionType.CreateRepos} />;

  case ActionType.CREATE_SCAN:

    return <BackendActionView data={data} actionType={BackendActionType.CreateScan} />;

  case ActionType.GRAPH:

    return <GraphRender result={result} context={context}/>;

}

And in order to better present the technical-related graphics details, we introduced a fifth (due to the existence of several graphics libraries, the construction became a pain, probably the biggest technical debt): Mermaid. The previous Echart.js can provide us with low-cost graphics writing, and D3.js allows for more versatile customization.

Finally, try deploying

After we wrote the PoC and tagged it with confidence, we found that the automatically built Docker image didn’t work, which was in the middle of the night. Finally, to sum up, there are two reasons:

• WebSocket for Nginx is not configured.
• The Kotlin REPL depends on the unpack environment.

Fortunately, an easy fix and a tag will suffice. Facts have proved that if you want a quick fix, you can’t a quick fix.

Configure WebSockets

First, according to the online documentation, configure the corresponding WebSocket:

location /ascode {

  proxy_pass http://archguard-backend:8080;

  proxy_http_version 1.1;

  proxy_set_header Upgrade $http_upgrade;

  proxy_set_header Connection $connection_upgrade;

}

Therefore, after rebuilding and building the image, it was found that there was a problem with the backend again, and the running REPL environment was wrong.

Configure Kotlin REPL classpath

As mentioned above, what the REPL configures in the code is:

val property = System.getProperty("java.class.path")

var embeddedClasspath: MutableList<File> = property.split(File.pathSeparator).map(::File).toMutableList()

However, after Spring is packaged, there is only one classpath, and Kotlin Scripting will have a series of problems, which is needed at this time requiresUnpack. For further information, consult the Spring Gradle plugin documentation: Just the “Spring Boot Gradle Plugin Reference Guide” corresponding explanation: the list of libraries that must be decompressed from the fat jars to run. Each library should be specified as having and of; they will be unpacked at runtime.

In effect, when Spring is running, it will extract the corresponding library from BootJar to a temporary directory.

tasks.withType<KotlinCompile> {

    kotlinOptions {

        jvmTarget = "1.8"

        freeCompilerArgs = listOf("-Xjsr305=strict")

    }

}




tasks.withType<BootJar> {

    requiresUnpack("**/kotlin-compiler-*.jar")

    requiresUnpack("**/kotlin-script-*.jar")

    requiresUnpack("**/kotlin-jupyter-*.jar")

    requiresUnpack("**/dsl-*.jar")

}

Of course, finding one by one is too difficult in terms of code, but I located the temporary directory and navigated it:

val tempdir = compiler[0].parent

embeddedClasspath = File(tempdir).walk(FileWalkDirection.BOTTOM_UP).sortedBy 
{ it.isDirectory }.toMutableList()

Finally, the generated classpath value looks like this:

ikotlin - Classpath used in script: [/tmp/app.jar-spring-boot-libs-5edaa25c-496e-4eb0-b7d6-1118a8cc280f/kotlin-script-runtime-1.6.21.jar, /tmp/app.jar-spring-boot-libs-5edaa25c-496e-4eb0-b7d6-1118a8cc280f/kotlin-jupyter-kernel-0.11.0-89-1.jar, /tmp/app.jar-spring-boot-libs-5edaa25c-496e-4eb0-b7d6-1118a8cc280f/dsl-2.0.0-alpha.12.jar, /tmp/app.jar-spring-boot-libs-5edaa25c-496e-4eb0-b7d6-1118a8cc280f/kotlin-compiler-embeddable-1.6.21.jar, /tmp/app.jar-spring-boot-libs-5edaa25c-496e-4eb0-b7d6-1118a8cc280f/kotlin-jupyter-api-0.11.0-89-1.jar, /tmp/app.jar-spring-boot-libs-5edaa25c-496e-4eb0-b7d6-1118a8cc280f/kotlin-jupyter-lib-0.11.0-89-1.jar, /tmp/app.jar-spring-boot-libs-5edaa25c-496e-4eb0-b7d6-1118a8cc280f/kotlin-jupyter-shared-compiler-0.11.0-89-1.jar, /tmp/app.jar-spring-boot-libs-5edaa25c-496e-4eb0-b7d6-1118a8cc280f/kotlin-jupyter-common-dependencies-0.11.0-89-1.jar, /tmp/app.jar-spring-boot-libs-5edaa25c-496e-4eb0-b7d6-1118a8cc280f/kotlin-script-util-1.6.21.jar, /tmp/app.jar-spring-boot-libs-5edaa25c-496e-4eb0-b7d6-1118a8cc280f]

In this way, it can finally start correctly, and then hit a new tag: v2.0.0-alpha.12.

Summarize

Although we have released this beta version, it still has a number of areas for improvement, such as:

• DSL architecture design. Compared with Ktor ‘s DSL design and implementation, the ArchGuard DSL appears to have no design.
• DSL syntax design. Not finished yet.
• Dynamic frontend components.
• Smarter editor support. Such as intellisense, autofill, etc.

So, welcome to ArchGuard to build an architecture workbench: Github Link: WorkGuard