Source code filters

Every now and then I stumble upon code that is not UTF-8 and has DOS line endings. Build programs don't like that very much. So instead of waiting for builds to fail I like to check source code so now and then.

Find files that are not compatible with UTF-8 encoding

$ find . -path ./.git -prune -o -type f -exec file -i {} \;|awk -F'[:;]' '$3 !~ /charset=us-ascii|charset=utf-8|charset=binary/ ? enc=$1 : enc="" { print $3, $1  }'

Count the number of files that have dos line endings

$ find . -path ./.git -prune -o -type f -exec dos2unix -id {} \; |awk '{s+=$1} END {print s}'

QA for shellscripts

Sonarqube is the tool I use to measure quality of code. There is no plugin for bash scripts, but with running a custom script in Sonarqube testing of bash script quality can be done. Check out shellcheck: https://github.com/koalaman/shellcheck

Find old projects in Jira

To keep Jira clean can be real challenging. I prefer to use just a handful of workflows, a small set of statuses, a couple of permission schemes and so on. What is see a lot is that projects, when they are finished, are not properly cleaned. A project should finish with a clean Jira project, issues are either new or done and all boards are gone. To view all ancient boards is not something that is easily done in Jira itself. So get some database access (I like HeidiSQL for this task) and run a query:

SELECT p.pkey AS PROJECT_KEY, p.pname AS PROJECT_NAME, MAX(i.UPDATED) AS LAST_UPDATED FROM jiraissue i INNER JOIN project p ON p.ID = i.PROJECT GROUP BY p.pname, p.pkey ORDER BY LAST_UPDATED

This will give you three columns, with the project key, the name of the project and the last updated date of the most recent updated issue, sorted with the oldest at the top. I use this to find the people who were responsible for the projects and then help them to clean. This will leanly (5S) help us to make Jira more an organisational standard and is part of the first 5S step, sorting.

Orthogonality and the DRY Principle

This is an article fomr 2003. They make fair points that are still applicable to developers now.

1. All programming is maintenance; only the first ten minutes code is new, the rest is maintenance.
2. Don't repeat yourself; every piece of knowledge in the development of something should have a single representation.
3. Code generators; if you expect a lot of volatility, a code generator can be a good investment.
4. Orthogonality;  things that are not related conceptually should not be related in the system

Read the full article: http://www.artima.com/intv/dry.html

Deprecation

Transformation from monolithic stacks to continuous delivery takes an other approach. In a monolithic environment it is a good practise to deprecate certain parts of the software, but it is not mandatory, everything is upgraded at once. But if one is going to deliver several updates a month, week or preferably a day, there need's to be a model to let applications and API users adapt to the new features.

Deprecation is when certain parts of the code are deemed obsolete and are ready to be removed. Because a lot of applications and interfaces depend on those parts of the code, it is hard to remove them. Therefor they should be marked deprecated and removed in time. Changing code, by refactoring for instance should never change a function call. A call should always be backward and forward compatible. Adding new parameters to the call is fine, removing any of them is considered evil. Whenever something is being deprecated a end of life time should be communicated, being either a date or a version. A version number can also tell us about deprecated features, whenever a major version changes, expect a cleanup of deprecated functions. Minor version changes should never break code.

So far, this has been the easy part. Code is easy to maintain an deprecate. It get's harder when we switch to API's. Most API's carry a version number in their URI. Which means that there is no deprecation, and whenever a new version of the API is released on a new URI, it is allowed to break all protocols compared to the previous version. A better practise is to update the existing API with new methods, extra parameters and communicate about what interfaces will disappear in the
comming period. This allows clients of the API to make adjustments to their client software in a timeslot whenever it suits them best.

Database updates are the hardest in my experience. A lot of times things get really fuzzy at this point. Adding columns when upgrading, inserting new base data, perhaps some PLSQL is added or removed. In a monolithic model an upgrade and a rollback of software goes hand in hand. We need extra SQL to make rollbacks possible. In a continuous delivery environment there is no need, because we only roll forward. However, changes to SQL need to be versioned as well; not in a single big SQL script for all applications. But added to the mini-, micro- or macro-service, in the module. In many projects I find nicely separated modules but one big cloth of SQL. Add the SQL of a module to that module, so it's kept under the same version as the code. A base SQL to generate the major version of the database, and all updates as separate files.

So far this is a very small piece of text from me about deprecation; barely scratching the surface, but there are lot's of good websites explaining how it should work. GIYF

A note about SSL/TLS trusted certificate stores, and platforms

Why are there so many places on a Linux machine to put in certificates? AdamW has a nice explanation I don't want to hold out. In short: /etc/ssl is Debian bases, /etc/pki is RHEL based. There is no standard, and all do what they like.


Gratitude to AdamW: https://www.happyassassin.net/2015/01/12/a-note-about-ssltls-trusted-certificate-stores-and-platforms/

Flame it on me

The other day I was working for a company that hired me to upgrade their software factory to a Continuous Delivery setup. One of the questions of the developmentteams was to introduce Git as a replacement for SVN. Working for a fairly big organisation that is traditionaly formed I had to consult the Solution Architect. Solution Architects are people living in Ivory Towers who haven't written software for a long time or never made their hands dirty anyhow.

So I filed a request to setup some Git and Gitlab to let my development teams try it out and see if it would suit their needs. In the request we made clear what the advantages would and could be; including easier branching and better merging, and we thought we had a strong case.

Then we recieved the verdict and I quote: "It requires too much effort, and does not conform with our rules on branching, merging and versionmanagement. The transfer to Git doesn't support a clear goal. Branching and merging should be an exception instead of a rule".

[silence]

Where did these people live for the last eleven years, under a rock? Developers shouldn't branche? Merging is hard therefor we should not do it? No... merging is easy with Git therefor we shouldn't be afraid of it anymore. If you are afraid of something, do it more often. Branching is part of a good delivery strategy in build pipelines, which make you flexible. Too bad I'm not their manager, I would fire all architects that are not up to date with newest developments.

In the reply he also wanted to know how Git handles binaries in contrast to Subversion. OMG, realy? Binaries? Why? FFS? Yes, these files can be placed in Subversion or Git, but why would you do that? There are binary repositories that can handle these kinds of files way better than code repositories. Isn't that what we learned in kindergarten?

And as last argument he raises that it is already a problem to get non-functionals done within the teams. For one I'm doing the change, team impact is minor; they need to install git and pull. Second it seems that everyone always uses the non-functionals for exchange and to implement last. Non-functionals are most of the time acceptance criteria that can be automated, this should be implemented in the factory at the start of the project. Not used as change money.

Please let the solution architects find some solution to disolve in, I've had it with them.

Puppet

When operations needs an installation manual, wouldn't it be better to provide them with the exact instructions on how to install? Yes, we use Ansible or Puppet!

Linkdump for Hello World with Puppet: https://www.scriptrock.com/getting-started-with-puppet-labs

Managing Windows machines from Linux

To manage machines is one thing, but logging in to Windows machines with RDP seems a total waste of time to me. So I need something else. Just headless machines maintenance from a linux host with Python.

For communications with the Windows machines I will use wmic.
http://www.iggyt.org/doku.php?id=wiki:infrastructure_tools:wmi:wmi-linux_client-wmic

After installing with yum I make a virtual environment with my favourite user and install wmi-wrapper
https://github.com/kanzure/python-wmi-client-wrapper

Powershell unit-testing

Last week I started Powershell. As a luser I'm spoiled with the Linux command prompt and cram testing. Powershell is the same but different. Since I'm bound to the evil OS supporting Powershell I thought to myself, why not have a look into it. Batch-scripts are way too limited, echo'ing something to the screen is about as much as one can do with it. So Powershell is the only option.

After my first nifty script I wanted to test it. Tricky, no unit-test framework by default. So I stumbled in my search upon Pester. After a download (.zip) and some tweaking it's running! Looks great, needs some work, but it will do the job, go check it out: https://github.com/pester/Pester

Cmd & Bash

I don't have much text here, just a link dump for today:

The art of command line:
https://github.com/jlevy/the-art-of-command-line/blob/master/README.md

Bash strict:
http://redsymbol.net/articles/unofficial-bash-strict-mode/

RESTful essay

I never lost my interested in REST, I even more love how easy people can do it wrong. I'm dumping the link to this essay while reading it. 

In my definition, a real-world RESTful API is an API that provides answers to questions that you won’t find in introductory texts, but that inevitably surface in the real world, such as whether or not resources should be described formally, how to create useful and automatic command-line interfaces, how to do polling, asynchronous and other non-standard types of requests, and how to deal with operations that have no good RESTful mapping.

 https://restful-api-design.readthedocs.org/en/latest/intro.html

Microservice Architecture

Great article by Martin Fowler about the microservices architecture model.


Most application development efforts that we see use a project model: where the aim is to deliver some piece of software which is then considered to be completed. On completion the software is handed over to a maintenance organization and the project team that built it is disbanded.
Microservice proponents tend to avoid this model, preferring instead the notion that a team should own a product over its full lifetime. A common inspiration for this is Amazon's notion of "you build, you run it" where a development team takes full responsibility for the software in production. This brings developers into day-to-day contact with how their software behaves in production and increases contact with their users, as they have to take on at least some of the support burden.



Source: http://martinfowler.com/articles/microservices.html

Warning: No support for locale: en_US.utf8

Fix the locale warning in Linux Mint:
http://forums.linuxmint.com/viewtopic.php?f=42&t=111527

Update Linux Mint: http://www.tecmint.com/upgrade-linux-mint-16-to-linux-mint-17/

After update fix samba: sudo apt-get install gvfs-backends

More on CD with Jenkins, Vagrant and Docker

More on CD with Jenkins, Vagrant and Docker:

1. http://technologyconversations.com/2014/04/29/continuous-delivery-introduction-to-concepts-and-tools/
2. http://technologyconversations.com/2014/05/06/continuous-delivery-ci-tools-setup/
3. http://technologyconversations.com/2014/05/20/continuous-delivery-static-analysis/

And more to come...

Tnx Dennis!

Postmortem

What should we do when something goes terribly wrong?

how-to-run-a-postmortem-with-humans-not-robots-velocity

Programming Sucks

This is a world where people eschew sex to write a programming language for orangutans. All programmers are forcing their brains to do things brains were never meant to do in a situation they can never make better, ten to fifteen hours a day, five to seven days a week, and every one of them is slowly going mad.

See: http://stilldrinking.org/programming-sucks

Ansible for CD

I'm working on putting down a DTAP environment, but i want to skip the classic DTAP where there is a lot of manual work and time gained in a scrum process is wasted in the waterfall way of working.

So i need some automation there. Automating means that i want users to be able to go to a website, select (or create) an environment and install the software that should be tested.

I had a look at Puppet and set up a test between two VBox instances. Is working with not too much work, but i don't like it. Seems a bit of a steep learning curve.
Then someone pointed out Ansible and i started to watch the video from last PyConn: http://www.ansible.com/resources. I'm already convinced this is the tool i need. And i'm happy to set up a prototype implementation!

Python tricks

Sahand Saba has compiled a list of some 30 Python Language Features and Tricks You May Not Know About. Chances are you know some, most of them are very interesting!

http://sahandsaba.com/thirty-python-language-features-and-tricks-you-may-not-know.html

Continuous Delivery and Feature Toggle

So you want DTAP? Nice, but what about a Continuous Delivery (CD) environment? That will bring more speed in the delivery process, why wait till the end of a sprint to deliver when you can do it in an instant. Adding business value when you are done!

But what about all outstanding changes? When you want to avoid merging hell, there is another way to go, it's called Feature Toggle. Just keep a file with toggles (true or false) on functionality and business rules and only activate them when you are really done.

http://martinfowler.com/bliki/FeatureToggle.html