CI Design
Let’s not derail during engineering stages
One of the key areas that needs focus at all times is quality. Without good QA the defect rate rises due to uncertainty inferred from changes to the systems capability. It’s not just ensuring the solution builds – although that might be the first test – but performing analysis for optimisation and security.
Lot’s of benefit to be gained from a little effort up front. I’m a big fan of QA and DevOps as this will create a level playing field for anyone that wants to contribute.
Evolution is the process by which testing new capabilities either proves successful, or kills it. Is this capability operable and does it have value? And I don’t mean “…well it might have value if some company does ‘X’ in ten years…”.
Static Code Analysis
You can’t get all the problems via static code analysis, but you can potentially identify traits which may give clues about flaws in your code when it’s executing. I’m going to come back to the security analysis and focus on the coding standards analysis, expecting that two tool-sets will be needed which are focused on each particular job.
To get both in one and have both capabilities working well in one tool would be a win-win.
However Ringo Dingo throws another curve-ball in that I know that .NET Core, Python, HTML, ISO/IEC 9075:2003 (SQL:2003), SQL-99 and Javascript are all involved in the platform.
Code Quality Analysis
I’ve typically lent towards something like Fortify (which covers security and code-quality) or Checkmarx in enterprise-grade scenarios, but I think this calls for use of open-source approaches. VS Code already has substantial live code quality analysis tools with the C# plugins, so I don’t want to do too much in this category.
Tool | Language | Notes |
SonarSource Community | .NET | I’ve listed SonarLint in the security analysis section below, but it could be that the same tool-set covers quality too |
nDepend | .NET | Like the technical debt estimation aspects in here… if money was going to be forked out this would be near the top of the list |
TBC | .NET | Just remembered I’m going to need something like dotTrace for performance analysis and diagnosis |
GitLab | Multi | Ok so I searched for Fortify-esque tools and found a GitLab page describing how it’s Auto DevOps covers this already. |
Query Plan Analysis | SQL | Not so much a tool but use of RDBMS platform query & analysis. I’m not keen on building a toolset for GitLab CI from scratch, as I did for MS SQL in past years. There’s some notation in the SonarLint documentation, but that’s IDE-side. |
What’s clear here is that whilst I’ll need to look for a dotTrace equivalent for VS Code and Linux, the default CI can be configured in GitLab to do a large portion of the work.
The other core part of CI is running the unit tests successfully – something that should be done before code is pushed back into the working branch, but often skipped or forgotten.
For .NET it’s fairly simple right up until API call tests (perhaps that should fall into the integration testing rather than automated unit testing), and I need to look into Python and Javascript unit testing.
Security Analysis
I used the OWASP list of code analysis tools as a basis, focusing on .NET. Really this tool needs to be runnable as part of a solution build in VS Code, and somehow runnable as a build task job on the CI runner. That way contributors developing code could ignore security warnings / add their local suppression configuration, without affecting the end result – as GL CI will enforce the full rule set.
Of course we should also expect that code analysis tools are specific to language, and that we will need to configure the GitLab runners for each code base update. I’ve selected the following initial candidates for this category on a per language basis:
Tool | Language | Notes |
Bandit | Python | Looks like it’s maintained currently, and could give good coverage for issues in Python. Needs a road test |
Google Search Diggity | N/A | Think this may form part of the post deployment scrutiny rather than a codebase check. It’s focused on assessing a web application whilst running |
Security Code Scan | .NET | This looks like a good tool to road-test for use within the environment. Not sure if this will work on VS Code on Linux yet, though it has NuGet package installation instructions |
LGTM | Multi | Could be the CI tool at the GitLab runner end. Needs investigation |
Puma Scan | .NET | The server-side flavour is commercial so the end-user (community) edition needs careful comparison. Do I want to use a commercially-funded product which may later close off, but be audited and funded properly… or an open-source (SCS) alternative which is community dependant? |
SonarLint | .NET | Like the look of this – IDE-embedded live code analysis tool. Works with VS Code OotB. Maybe this for IDE and either SCS or Puma Scan server-side? It’s part of the overall SonarQube platform which was listed in the previous section |
I’ll also take a look at OWASP O2 for the overall Ringo Dingo piece.
Other Thoughts
For now I’m going to say it’s “OK” to add overrides for warnings on exceptions relating to coding standards such as curly-braces-on-newline, and other such stuff. I’m a K&R-style fan and don’t care much for that – I’m likely to make that a standard for the repository submissions as well. I’m also not interested in debating that aspect.
I’m initially going to add suppression for IDE0063, so that standard using statements are not producing warnings. Whilst it’s ‘easier’ to use the newer notation I also see many defects being created where someone forgets the scope of something (or mixes scope at the same indentation level unintentionally).
Conversely it’s not ok to add a suppression just to get a section of code working. Coding style is one thing but likely that particular warning or error was there for a reason.
The problem area is going to be code analysis for PostgreSQL. Suspect there will be custom grep scripts for GitLab CI which search for particular problem syntax, and the need to try and create unit testing for procs to ensure they query within performance limits. No small task.
If I build that capability, I’m definitely going to open-source it as the community will be able to use the framework to do a far better job than I can at this stage.
It’s also worth periodically reviewing why warnings or exceptions might be explicitly (configured) overridden. Something you might do in an early phase to get the component working might not be appropriate later, especially approaching production.
Definitely need to review all suppression and exemptions before pre-prod deployment, and will have to think about what mechanisms will help achieve that – perhaps something linked between development branches and sitting in-between dev and master branches? We’ll see – normally current version is master, with development branches for each feature group in development.
So much to consider.