I have posted in a previous article How to read someone else’s programming code in that article we mentioned how you can easily read someone’s code and also you can learn PyTorch’s code I explained how it’s easy and fun for code.
In this article, I will tell you how we build tools for API and architecture guarding rules and making KtLint and many more stays tuned.
In making this tool some factors to consider are:
- Plugin. Developers can develop some new architecture guarding rules based on the existing guarding rules, such as those for API and database call links.
- Testability. If a full DSL or semi-DSL is used, how can the subsequent
- Language is irrelevant. How to implement support for multiple languages without being bound to the syntax tree of the language.
For this purpose, I have to pick up the existing code and analyze it. There are mainly four tools, KtLint for Kotlin language, Spectral for OpenAPI, SQLFluff for multiple databases, and expressions such as MyBatis. style language Ognl.
Governance of Kotlin code: KtLint
The difference between KtLint and general Lint tools is that it comes with an automatic formatting function. The overall logic of KtLint is relatively simple. It generates AST based on a single file and then performs rule matching against the AST. Ktlint builds a hierarchical relationship of rules around Rules, Rulesets, and RulesetsProvider, and uses the Visitor (VisitorProvider) mode to analyze the AST. The following is the abstract Rule of KtLint:
/** * This method is going to be executed for each node in AST (in DFS fashion). * * @param node AST node * @param autoCorrect indicates whether rule should attempt auto-correction * @param emit a way for rule to notify about a violation (lint error) */ abstract fun visit( node: ASTNode, autoCorrect: Boolean, emit: (offset: Int, errorMessage: String, canBeAutoCorrected: Boolean) -> Unit )
As said in the comments, the three parameters represent their respective purposes. Here ASTNode is an AST tree ( kotlin-compiler-embeddablepackage) derived from Kotlin. The mode is also to get the configuration, and then run the detection rules:
val ruleSets = ruleSetProviders.map { it.value.get() }
val visitorProvider = VisitorProvider(ruleSets, debug)
The corresponding visit:
visitorProvider
.visitor(
params.ruleSets,
preparedCode.rootNode,
concurrent = false
).invoke { node, rule, fqRuleId -> }
The corresponding rules are filtered in the VistorProvider:
val enabledRuleReferences =
ruleReferences
.filter { ruleReference -> isNotDisabled(rootNode, ruleReference.toQualifiedRuleId()) }
val enabledQualifiedRuleIds = enabledRuleReferences.map { it.toQualifiedRuleId() }
val enabledRules = ruleSets
.flatMap { ruleSet ->
ruleSet
.rules
.filter { rule -> toQualifiedRuleId(ruleSet.id, rule.id) in enabledQualifiedRuleIds }
.filter { rule -> isNotDisabled(rootNode, toQualifiedRuleId(ruleSet.id, rule.id)) }
.map { rule -> "${ruleSet.id}:${rule.id}" to rule }
}.toMap()
....
Then, run the visits in the Rule in parallel or serially.
The way of rules is the plug-in way through ServicesLoader:
private fun getRuleSetProvidersByUrl(
url: URL?,
debug: Boolean
): Pair<URL?, List<RuleSetProvider>> {
if (url != null && debug) {
logger.debug { "JAR ruleset provided with path \"${url.path}\"" }
}
val ruleSetProviders = ServiceLoader.load(
RuleSetProvider::class.java,
URLClassLoader(listOfNotNull(url).toTypedArray())
).toList()
return url to ruleSetProviders.toList()
}
Wouldn’t it be more convenient to use Java 9 modules if the granularity was larger?
Spectral based on API data
Unlike Ktlint, Spectral is a tool for JSON/YAML Lint, especially OpenAPI documentation (that is, swagger’s yaml/json files). Compared with Ktlint, the most interesting part of Spectral is that it provides a JSON Path (similar to XPath) function, which can apply specific rules to specific parts of objects. Here is an example of Spectral:
'oas3-valid-schema-example': {
description: 'Examples must be valid against their defined schema.',
message: '{{error}}',
severity: 0,
formats: [oas3],
recommended: true,
type: 'validation',
given: [
"$.components.schemas..[?(@property !== 'properties' && @ && (@ && @.example !== void 0 || @.default !== void 0) && (@.enum || @.type || @.format || @.$ref || @.properties || @.items))]",
"$..content..[?(@property !== 'properties' && @ && (@ && @.example !== void 0 || @.default !== void 0) && (@.enum || @.type || @.format || @.$ref || @.properties || @.items))]",
"$..headers..[?(@property !== 'properties' && @ && (@ && @.example !== void 0 || @.default !== void 0) && (@.enum || @.type || @.format || @.$ref || @.properties || @.items))]",
"$..parameters..[?(@property !== 'properties' && @ && (@ && @.example !== void 0 || @.default !== void 0) && (@.enum || @.type || @.format || @.$ref || @.properties || @.items))]",
],
then: {
function: oasExample,
functionOptions: {
schemaField: '$',
oasVersion: 3,
type: 'schema',
},
},
}
The in the above givenobject is for the relevant properties in the object as a condition to execute the following thenfunction. For details, please refer to the official document: ” Custom Rulesets “. By the way: Spectral uses nimma as JSON Path expression.
Spectral’s model
Compared with Ktlint, Spectral’s data model is slightly more complicated because it is related to OpenAPI/Async API binding, plus specific regular expressions. Its data model includes description, message level, given-then, and context. As follows:
- recommended. Is it a recommended configuration.
- enabled. Is it allowed
- description. Rule description
- message. error message
- documentationUrl. document address.
- severity. Severity, `error`, `warn`, `info`, or `hint`.
- formats. Formatting standards such as OpenAPI 2.0, OpenAPI 3.0, etc.
- resolved. Is it resolved.
- given. Similar to selectors in CSS, using JsonPath similar to XPath, JSONPath
- then。
- field, field
- function, function, pattern
- functionOptions
In addition, it has a simple type system and corresponding expression judgments. as follows:
- CASES。flat、camel、pascal、kebab、cobol、snake、macro
- Length: Max, Min.
- number
- Boolean judgment.
- type system. enumerate
Overall, Spectral is flexible and interesting to implement.
SQLFluff
SQLFluff is more challenging for applications based on existing data models like Ktlint and Spectral – it builds rules based on many different database dialects. SQLFluff analyzes directly based on the source code, converting different database dialects into basic elements (word segmentation). Then, they are processed based on the type + rules of tokenization. In short, it is a more abstract word segmentation context to construct the corresponding rule context. as follows
- segement. At its core is BaseSegment, which defines the three basic elements of Lexing, Parsing and Linting, and produces such as: groupby_clause, orderby_clause, and select_clauseother participles.
- parent_stack。
- siblings_pre。
- siblings_post。
- raw_stack。
- memory。
- dialect. as the basis for runtime parsing of the grammar.
- path. path.
- templated_file. template file.
Example:
{ "file": {
"statement": {
"select_statement": {
"select_clause": {
"keyword": "SELECT",
"whitespace": " ", "select_clause_element":
{ "column_reference": {
"identifier": "foo"
}
}
},
"whitespace": " ",
"from_clause": { "keyword": "FROM", "whitespace": " ",
"from_expression": {
"from_expression_element": {
"table_expression": {
"table_reference": {
"identifier": "bar"
}
}
}
}
}
}
},
"statement_terminator": ";",
"newline": "\n"
}
}
Subsequent rules are performed on these rules eval, as shown in the following example:
class Rule_L021(BaseRule):
def _eval(self, context: RuleContext) -> Optional[LintResult]:
"""Ambiguous use of DISTINCT in select statement with GROUP BY."""
segment = context.functional.segment
if (
segment.all(sp.is_type("select_statement"))
# Do we have a group by clause
and segment.children(sp.is_type("groupby_clause"))
):
# Do we have the "DISTINCT" keyword in the select clause
distinct = (
segment.children(sp.is_type("select_clause"))
.children(sp.is_type("select_clause_modifier"))
.children(sp.is_type("keyword"))
.select(sp.is_name("distinct"))
)
if distinct:
return LintResult(anchor=distinct[0])
return None
All rule judgments here are based on this abstract syntax tree. In a sense, a unified abstraction is constructed. I wanted to analyze further but found that there are various regular expressions in various SQL dialects, so I chose to retreat temporarily.
Expression Language: OGNL
At first, when I implemented ArchGuard Scanner’s SQL generation support for MyBatis, I saw OGNL expressions nested in XML and discovered OGNL. In terms of implementation, it is more sophisticated in combining with data than the S-expressions in TreeSitter that I envisioned earlier. Similarly, it can also be used for rule judgment here, and expressions can be used to match data.
Object Graph Navigation Language, or OGNL for short, is an open-source Expression Language used in Java to get and set properties of Java objects, as well as other additional functions such as list projection ( projection) and the selection and lambda expressions. You can use the same expression to get and set the value of a property. The Ognl class contains shortcuts for evaluating OGNL expressions. It can do this in two stages, parsing the expression into an internal form and then using that internal form to set or get the value of the property; or it can do it in one stage and get or set the property directly using the string form of the expression.
Ognl.getValue("name='jerry'", oc, oc.getRoot());
String name2 = (String) Ognl.getValue("#user1.name='jack',#user1.name", oc, oc.getRoot());
Originally wanted to imitate OGNL to write an expression language, but found that Jacc was used, and there was no Antlr implementation. So, looking for a more reasonable way.
You may like it :
- Designing for Maintenance: The First Principles of Architectural Design
-
How Internet website architecture upgraded by changes in programming languages
2 Comments