Source code filters

Every now and then I stumble upon code that is not UTF-8 and has DOS line endings. Build programs don't like that very much. So instead of waiting for builds to fail I like to check source code so now and then.

Find files that are not compatible with UTF-8 encoding

$ find . -path ./.git -prune -o -type f -exec file -i {} \;|awk -F'[:;]' '$3 !~ /charset=us-ascii|charset=utf-8|charset=binary/ ? enc=$1 : enc="" { print $3, $1  }'

Count the number of files that have dos line endings

$ find . -path ./.git -prune -o -type f -exec dos2unix -id {} \; |awk '{s+=$1} END {print s}'

QA for shellscripts

Sonarqube is the tool I use to measure quality of code. There is no plugin for bash scripts, but with running a custom script in Sonarqube testing of bash script quality can be done. Check out shellcheck: https://github.com/koalaman/shellcheck

Find old projects in Jira

To keep Jira clean can be real challenging. I prefer to use just a handful of workflows, a small set of statuses, a couple of permission schemes and so on. What is see a lot is that projects, when they are finished, are not properly cleaned. A project should finish with a clean Jira project, issues are either new or done and all boards are gone. To view all ancient boards is not something that is easily done in Jira itself. So get some database access (I like HeidiSQL for this task) and run a query:

SELECT p.pkey AS PROJECT_KEY, p.pname AS PROJECT_NAME, MAX(i.UPDATED) AS LAST_UPDATED FROM jiraissue i INNER JOIN project p ON p.ID = i.PROJECT GROUP BY p.pname, p.pkey ORDER BY LAST_UPDATED

This will give you three columns, with the project key, the name of the project and the last updated date of the most recent updated issue, sorted with the oldest at the top. I use this to find the people who were responsible for the projects and then help them to clean. This will leanly (5S) help us to make Jira more an organisational standard and is part of the first 5S step, sorting.

Orthogonality and the DRY Principle

This is an article fomr 2003. They make fair points that are still applicable to developers now.

1. All programming is maintenance; only the first ten minutes code is new, the rest is maintenance.
2. Don't repeat yourself; every piece of knowledge in the development of something should have a single representation.
3. Code generators; if you expect a lot of volatility, a code generator can be a good investment.
4. Orthogonality;  things that are not related conceptually should not be related in the system

Read the full article: http://www.artima.com/intv/dry.html

Deprecation

Transformation from monolithic stacks to continuous delivery takes an other approach. In a monolithic environment it is a good practise to deprecate certain parts of the software, but it is not mandatory, everything is upgraded at once. But if one is going to deliver several updates a month, week or preferably a day, there need's to be a model to let applications and API users adapt to the new features.

Deprecation is when certain parts of the code are deemed obsolete and are ready to be removed. Because a lot of applications and interfaces depend on those parts of the code, it is hard to remove them. Therefor they should be marked deprecated and removed in time. Changing code, by refactoring for instance should never change a function call. A call should always be backward and forward compatible. Adding new parameters to the call is fine, removing any of them is considered evil. Whenever something is being deprecated a end of life time should be communicated, being either a date or a version. A version number can also tell us about deprecated features, whenever a major version changes, expect a cleanup of deprecated functions. Minor version changes should never break code.

So far, this has been the easy part. Code is easy to maintain an deprecate. It get's harder when we switch to API's. Most API's carry a version number in their URI. Which means that there is no deprecation, and whenever a new version of the API is released on a new URI, it is allowed to break all protocols compared to the previous version. A better practise is to update the existing API with new methods, extra parameters and communicate about what interfaces will disappear in the
comming period. This allows clients of the API to make adjustments to their client software in a timeslot whenever it suits them best.

Database updates are the hardest in my experience. A lot of times things get really fuzzy at this point. Adding columns when upgrading, inserting new base data, perhaps some PLSQL is added or removed. In a monolithic model an upgrade and a rollback of software goes hand in hand. We need extra SQL to make rollbacks possible. In a continuous delivery environment there is no need, because we only roll forward. However, changes to SQL need to be versioned as well; not in a single big SQL script for all applications. But added to the mini-, micro- or macro-service, in the module. In many projects I find nicely separated modules but one big cloth of SQL. Add the SQL of a module to that module, so it's kept under the same version as the code. A base SQL to generate the major version of the database, and all updates as separate files.

So far this is a very small piece of text from me about deprecation; barely scratching the surface, but there are lot's of good websites explaining how it should work. GIYF

A note about SSL/TLS trusted certificate stores, and platforms

Why are there so many places on a Linux machine to put in certificates? AdamW has a nice explanation I don't want to hold out. In short: /etc/ssl is Debian bases, /etc/pki is RHEL based. There is no standard, and all do what they like.


Gratitude to AdamW: https://www.happyassassin.net/2015/01/12/a-note-about-ssltls-trusted-certificate-stores-and-platforms/

Flame it on me

The other day I was working for a company that hired me to upgrade their software factory to a Continuous Delivery setup. One of the questions of the developmentteams was to introduce Git as a replacement for SVN. Working for a fairly big organisation that is traditionaly formed I had to consult the Solution Architect. Solution Architects are people living in Ivory Towers who haven't written software for a long time or never made their hands dirty anyhow.

So I filed a request to setup some Git and Gitlab to let my development teams try it out and see if it would suit their needs. In the request we made clear what the advantages would and could be; including easier branching and better merging, and we thought we had a strong case.

Then we recieved the verdict and I quote: "It requires too much effort, and does not conform with our rules on branching, merging and versionmanagement. The transfer to Git doesn't support a clear goal. Branching and merging should be an exception instead of a rule".

[silence]

Where did these people live for the last eleven years, under a rock? Developers shouldn't branche? Merging is hard therefor we should not do it? No... merging is easy with Git therefor we shouldn't be afraid of it anymore. If you are afraid of something, do it more often. Branching is part of a good delivery strategy in build pipelines, which make you flexible. Too bad I'm not their manager, I would fire all architects that are not up to date with newest developments.

In the reply he also wanted to know how Git handles binaries in contrast to Subversion. OMG, realy? Binaries? Why? FFS? Yes, these files can be placed in Subversion or Git, but why would you do that? There are binary repositories that can handle these kinds of files way better than code repositories. Isn't that what we learned in kindergarten?

And as last argument he raises that it is already a problem to get non-functionals done within the teams. For one I'm doing the change, team impact is minor; they need to install git and pull. Second it seems that everyone always uses the non-functionals for exchange and to implement last. Non-functionals are most of the time acceptance criteria that can be automated, this should be implemented in the factory at the start of the project. Not used as change money.

Please let the solution architects find some solution to disolve in, I've had it with them.