Showing posts with label links. Show all posts
Showing posts with label links. Show all posts

9 June 2016

Oracle Code QA

As Code Cop I want all my code to be clean so I keep my sanity when maintaining it. Some basic pillars that support internal code quality regardless of programming language are Coding Conventions, automated (unit) tests, Static Code Analysis and Continuous Integration. I discuss all of them in my Code Quality Assurance lecture (and its latest slides are here). A good development process covers all these and more.

Recently a colleague inherited a bunch of Oracle PL/SQL code and asked me for help. Being used to Java and many tools that help us keeping the code in shape, e.g. JUnit, Checkstyle, PMD, Jenkins, he wanted the same for his database code. While some programming language ecosystems are traditionally strong in supporting the things I mentioned earlier, some other languages seem to lack behind. Clearly there are fewer options for less used languages. But that must not stop us from applying the same rigour to our code. Let's get started!

Everyone is Marcin Grzejszczak todayDatabase Naming Conventions
First we need coding conventions because consistency is important. Unlike Java where most projects follow the Oracle conventions, there is no such thing for Oracle databases. Instead there are several, sometimes contradicting proposals and you have to put together your own set of rules. Here are some reasonable ones for schema objects:PL/SQL Coding Conventions
The Procedural Language/Structured Query Language (PL/SQL) was introduced by Oracle in 1992. It is a compiled, procedural and structured language. By these attributes it is similar to modern languages like Java or C#, and all the general advice for naming, formatting, commenting, function scope and code size apply. Even object oriented concepts like Encapsulation or Coupling are meaningful (to a certain degree). See my presentation on Clean PL/SQL for more details. Again there are no official conventions from Oracle.
  • Steven Feuerstein's Naming Conventions and Coding Standards contain a list of naming conventions for PL/SQL variables together with some guidelines and a discussion of rejected conventions. If you do not know Steven, he is probably the authority on PL/SQL programming and knows what he is talking about. He also outlines a way to check the conventions, which I really like.
  • Philip Greenspun's SQL Style contains a few rules on formatting SQL statements for better readability.
  • Trivadis' PL/SQL and SQL Coding Guidelines are a complete set of standards regarding naming, formatting, language usage and control structures. It is a very comprehensive document of almost 60 pages and looks really impressive.
How to Choose Your Own Conventions
As there is no standard, you need to roll your own. To get started I recommend reading all the resources above (and even google for some more) and get an idea what could and should be defined. Then you look at your existing database objects and source code. Usually developers follow some conventions and some percentage of the code uses similar patterns in formatting or naming. If one of the used conventions is in the limits of the different proposals above - and you like it - then start with it. (Starting from something that is already there reduces your options and the resulting conventions are less optimal, but on the other hand you have a bigger change to get the code into a consistent state, because some part of the code follows the rules. If there are no existing patterns in your code, if you are starting from scratch or if all you see is crap, you still need to define different conventions.)

Quality ControlStart with a small set of rules in the beginning. There should be some naming schemes, table aliases and formatting rules. If you define too many rules at once, there will be too many violations in the existing code and people will argue that adhering to the conventions is too much work. Later, when everybody got used to the rules, it is time to add more of them. You will find more specific rules during the lifetime of a project, e.g. by identifying bug patterns to be avoided in the future. Conventions need to grow. Unless you are beginning a new project and want to start with a full set of conventions, the Trivadis conventions mentioned above might be too comprehensive to start with. But they are an excellent example how a full blown conventions document looks like.

Reviewing your code, reading the provided resources and collecting the basic rules that apply and that you like should not take you more than a few hours. It is more important to start with the first version of coding conventions sooner than to start with a complete set later.

Unit Test
If you are used to JUnit, RSpec or Jasmine you will be disappointed. There is not much support for unit testing in PL/SQL.
  • Usually - if at all - developers create stored procedures that call other procedures and check the results programmatically. If these test procedures follow a common convention, e.g. raising an exception on test failure, it is possible to automate calling them from the command line or build server.
  • Another option is utPLSQL, a basic unit testing framework created by Steven Feuerstein. It works as expected, but lacks the comfort of modern unit testing frameworks. I used it to test my PL/SQL port of Gilded Rose. (Gilded Rose is a testing kata where you need to create a lot of tests. It is an excellent exercise to get a first impression of a unit testing framework.)
  • Oracle SQL Developer has some support for automated testing. Unlike utPLSQL it is driven through the user interface. Tests are created and executed through the UI of SQL Developer. A Test case is a set of input values - usually rows in one or more tables - and a call to a stored procedure. Then the updated values are compared against a set of expected values. To see this in action, check out Jeff Smith's introduction to Unit Testing Your PL/SQL with Oracle SQL Developer. It is easy to create first tests, but test definition lacks the power of a general purpose programming language. Further I do not like that test definitions are "hidden" in some SQL Developer specific tables, the Unit Test Repository. However if you are a heavy user of SQL Developer, it might still be reasonable to use it for testing, too.
  • DbFit is an extension to FitNesse, a standalone, acceptance testing framework. DbFit tests are written using tables, making some test scenarios more readable than xUnit-style tests.
  • Steven has more recommendations for unit testing PL/SQL, including Toad's Code Tester for Oracle. If you licenced the Toad Suite, it is worth checking out its testing functionality.
Unit testing is mandatory, but no single approach or framework looks superior. I believe the best approach is to evaluate the different options, using the tools you already have. Maybe create some tests for the Gilded Rose in each of the testing frameworks to see what works for you and what not.

Static Code Analysis
magnifyA vital part of code quality assurance is static code or program analysis. The source or object code is analysed without actually executing it to highlight possible coding errors. I love static analysis but have never used tools for PL/SQL myself. Steven agrees with me that we should use Lint Checkers for PL/SQL - and of course he is right. He lists some tools that add warnings besides the checks provided by Oracle.
  • A free tool is PMD for PL/SQL. PMD is my favourite code checker for Java and it works great. There are only a few rules for PL/SQL but more can be added easily. (See how I added custom rules to PMD in the past.) PMD is definitely a tool I would use first.
  • Trivadis PL/SQL Cop looks very promising. I am not sure about its licence, but it seems to be free. Rules must be checked automatically each day, e.g. in the nightly build, so tools must work from the command line. Again I do not know if PL/SQL Cop works like that. The next step would be to experiment with it and see if it can be run from the command line.
  • Another great tool is the Sonar Source PL/SQL Plugin. The plugin adds PL/SQL support to SonarQube. SonarQube is a free, open platform to manage code quality. It is widely used in the Java community. The plugin is commercial, but if you need to manage a lot of PL/SQL code I would recommend buying it nevertheless.
  • There are several other commercial tools available, e.g. ClearSQL by CONQUEST, which I did not check.
For static code analysis I follow the rule that more is better. I recommend to start with a basic tool, e.g. PMD, and keep adding tools and rules over time. In existing projects you need time to fix the violations, e.g. WHEN OTHERS THEN NULL, and starting with too many rules in the beginning creates a lot of work.

Putting It All Together
After you established conventions, added automated tests and configured some static analysis tools, it is time to put it all together and shorten the feedback loop. While you could run tests and checks manually from time to time, it would be more helpful to do so every night, or even better on each check-in/push. (Checking your code on each check-in requires you to put your DDLs and package sources under version control. While this adds some extra steps to your development workflow, I highly recommend doing so.) A tool like Jenkins or another Continuous Integration server could be used to create an empty database instance, execute your DDLs and compile all your packages. Starting with an empty database instance is important to avoid works on my machine problems. Then Jenkins should run all tests to verify that the code works as expected. The final step is to analyse your code for violations of coding convention and potential problems. Many people add more steps like generating documentation or packaging deployment bundles suitable to be deployed by the operations team's DBA.

Just Do It!
Terence Parr recommends to automate anything that you might screw up and he is right. Creating working software is hard enough, we should not bother with manual tasks, rather automate them. Further automated checks keep the quality of our software high, resulting in faster maintenance and less bugs. This leaves us more time for the interesting parts of software development - solving problem and creating solutions.

4 December 2012

Eclipse Plugin Development

My friend Piotr asked me where to start with Eclipse/RCP development. I am not an expert on Eclipse but it is the main platform of my current employer, and I collected some information while digging deeper into the topic myself. Following an advice from Scott Hanselman I write this blog post instead of an email. This is my list for developers starting with Eclipse development.Eclipse
  • The first pages to read are the Eclipse Plugin and Eclipse RCP Tutorials by Lars Vogella. These are short tutorials with good content which are highly relevant. You might want to start here.

  • Another source for tutorials is G. Prakash's Eclipse Tips, especially his top ten mistakes in Eclipse Plug-in development are highly recommended.

  • More details can be found in the book about Eclipse, Eclipse Rich Client Platform by Jeff McAffer, Jean-Michel Lemieux and Chris Aniszczyk. Throughout the book the authors build an entire application, set up the automated build, create an update site and everything else. This is a comprehensive end to end description.

  • The Eclipse website itself hosts a lot of specialized information. My favourite article is about how to use the JFace Tree Viewer. This does not only show how to use the Tree Viewer, it also explains how to "think in JFace" in general.

  • Finally a lot of articles can be found on IBM developerWorks, just search for RCP. These articles are older, mainly from 2006 to 2008, but most things discussed there are still relevant.
That should be more than enough to get you started ;-)

6 October 2012

Design Lessons

ClassroomJust finished my weekly cycle workout. As usual I watched a presentation, this time it happened to be Deep Design Lessons by Michael "Legacy Code" Feathers, recorded at NDC Oslo 2012. In this talk he outlines some of the gems that he feels are less understood today. You really need to see this. There is so much to learn from his few slides. Now go, watch it! (Thanks to Claus for sending me the link.)

When writing about talks by Michael Feathers, I always need to mention my all-time favourite Deep Synergy Between Testability and Good Design. I am repeating myself, but I highly recommend watching it. It is an excellent talk examining the relationship among test-pains, code smells and design principles.

If you want to follow what I watch, here are the presentations I liked.

11 August 2012

Remote Pair Practice

Deep in my notes of past conferences I keep a quote that says Writing code together is the only true way that programmers communicate. Even if you do not agree, pair programming is still a great way to share with one another. Whenever I program in a pair, I learn a lot. Unfortunately I have a very little opportunity to do so as my team is distributed around the globe. After attending a Test Driven Development workshop at this year's GeeCON I decided to do something about it.

Pair Programming for AilurophilesSchedule
I plan for a pair coding activity each week. This seems like a good target to aim for. To free one or two hours each week for deliberate practice should be possible for everybody. Unfortunately I do not reach my target because I might be busy, might still look for a pair or my pair might be busy as well. In the past I only managed to participate in a pairing session twice a month. By writing this post I hope to save time on explaining and to improve the frequency of my pair programming. I found it better to schedule throughout the day instead of the evening. It also seems easier to find a suitable time at work days instead of weekends. Keeping away from evenings and weekends helps because there are still people with some private life ;-). An appointment at a fixed day and time is an option when pairing with the same person again and again, but to meet with different people, one has to be flexible. I do not care for the time as long as it is somewhere in my (European) time zone. To find suitable times in advance I use Doodle.

Pair Programming
A lot of people are afraid of pair programming because they never did it. To get started you should read James Shore's description of Pair Programming from his book The Art of Agile Development. There is even a short summary consisting of 99 words for the people who do not have time to read. In the end the rules how to behave during pair programming, are about practising everyday civility, because all I really need to know about pair programming I learned in Kindergarten. So be especially courteous. Pair programming teaches us to collaborate. I recommend watching Angela Harms' presentation about the (social) anti patterns in pair programming, Does pair programming have to suck? Note that pair programming may be difficult and uncomfortable for the first few times.

Get-Together
Obviously we have to meet in order to pair. In the past I managed a few times to meet during lunch breaks and have a short pairing session, much like Code & Coffee. Now I always bring my laptop when I meet my friends for lunch, maybe he is interested in a spontaneous pairing session while eating?

Remote Setup
Meeting in person is preferable but not always possible. Remote pairing is mainly about the additional tooling and all rules from regular pairing apply as well. My friend Thomas wrote a good post about Pair Programing from which I took all the material of this section. We use Skype to talk and TeamViewer to share our screens. It helps to have Skype contacts exchanged in advance so one can start the remote session by sending a chat message to the other one. As soon as the line is established we discuss who will host the session. A good headset is recommended, otherwise the voice quality is poor.

Then the host starts his TeamViewer and sends his TeamViewer id in Skype. I never had any problems with TeamViewer and it works great, even when connecting Windows and Mac machines. TeamViewer is a commercial tool but free for non-commercial use. A pairing session is definitely for educational purposes, so I do not see any problems with the license. After connecting I recommend changing the resolution of the host machine to the maximum resolution supported by the client. As a client I use TeamViewer's options View > Original and and View > Full Screen to get rid of all these scroll bars. If I have an additional screen I put the Skype window and a browser window there, if I want to look something up quickly. This setup is much like Level Four Pair-adise, except that it is remote.

It happens that I would like to scribble something on a white board. There is a nice Web Whiteboard, which allows drawing and shared editing. It is as simple as a real white board. If you need something more advanced, create a new Scribble in Google Docs which allows shared editing as well. All these and even more advanced topics of remote pairing are discussed by Joe Moore in his excellent talk about Remote Pair Programming: People and Technology.

A pairCode Kata
An educational pairing session looks much like a Coderetreat. We might do a code kata or whatever coding problem seems adequate to work on. A code kata is an exercise in programming which helps a programmer improve his or her skills through practice and repetition. When I meet someone new, I do not impose any special rules. We would just work on a kata, preferably one that we both know, using ping-pong styled Test Driven Development. Todd Sedano compiled a list of code katas sorted by how easy it is to apply TDD. Fizz Buzz and Prime Factors are too short, but all other katas beginning with String Calculator are suitable. In fact the size of the kata does not matter, because it is not about finishing, it is about improving.

The choice of programming language is not important to me. Although my primary language is Java, I have done katas in Ruby, Scala, JavaScript and C# as well as some other, less common languages. After choosing the kata, the language and the development environment to use, we need to agree on the focus of the training session. There are some variations of katas possible, e.g. to use a more functional programming style or not to us any primitives or conditionals, up to the (at least for me) extremely difficult TDD as if you meant it.

With Tomatoes
It happened that we would lose track of the time while focusing on the problem. Later, for example after working on the kata for two hours, we would "wake" up and be totally exhausted. To prevent this, and to create some space for chit-chat, I use the Pomodoro Technique. There is a break of 5 minutes every 25 minutes. I just watch out to stop the Pomodoro with a failing test, so we know where to continue after the break.

Conclusion
As shown my remote pair practice contains everything that is good and holy: pair programming, cool tools like video conferencing and screen sharing, coding, TDD, code katas and Pomodoros. Can you resist?

26 April 2012

JUnit Tutorials

I prepared a short list of tips and links for some colleagues to get started with JUnit.
  • A good start for an absolute beginner is the JUnit Cookbook.
  • JUnit is integrated in Eclipse. Lars Vogel's JUnit Tutorial shows how to write and execute tests using Eclipse. Talking about Eclipse, I always use the keyboard short-cut ALT-SHIFT-X and then key T to launch the test open in the editor window. After that I use the keyboard short-cut CTRL-F11, which runs the last launched test or application, to re-run my JUnit test until it succeeds.
  • JUnit comes pre-packaged with Hamcrest, a framework for writing matcher objects. These matchers improve the readability of tests and provide better failure messages. Consider writing a custom Matcher if you need to compare large objects with one another based on complex state.
  • A mistake that I see quite often is the handling of expected exceptions in tests. Szczepan Faber has written down five rules how to do that and to avoid any problems.
  • To explore more advanced topics of JUnit see my presentation on Practical Unit Testing (June 2009).
  • Also make sure that your tests are deterministic. Non-deterministic tests are a serious threat to the discipline of unit testing.

Ventanas Rotas. Broken WindowsAll these links focus on using the JUnit technology, but do not explain how to write good tests. If you have some spare time and would like to know more about JUnit and unit testing in general I recommend the book Pragmatic Unit Testing in Java with JUnit.

Behaviour - the "New" Way
By the rise of BDD the common understanding of unit tests has changed. Even if you do not do TDD or BDD these things apply to any unit test.
  • They are now called Micro tests to distinguish them from traditional unit tests. Tests with a larger scope, e.g. integration or end-to-end tests, are no unit tests even if they make use of JUnit.
  • Further the names of test methods should be full sentences focusing on behaviour. JUnit 4.x removes the need to prefix test methods with test, and usually the sentence of the expected behaviour starts with the word should.
  • The test methods should be created using the Arrange-Act-Assert or even better the Given-When-Then pattern.

Recommended Watching
Finally I highly recommend the recording of The Deep Synergy Between Testability and Good Design by Michael Feathers. It's an excellent talk examining relationship among test-pains, code smells and design principles. Go, watch it!

13 February 2010

Testing For All One's Worth

In the end of last year, after a long break, the third part of the 'Code Cop' series has been published in the well known German magazine iX:

Crash DummyTägliche Builds mit automatisierten Tests (Daily Builds with Automated Testing) (iX 1/2010). [... Automated testing is vital for the quality assurance. Unit-tests are applied easily using JUnit. The same is true for functional testing thanks to a number of already existing tools. By adding testing capabilities to the build, developers are more willing to write tests. In the end the analysis of the code coverage achieved by the tests reveals some possible weak points. ...]

(Download source code of Ant/JUnit/HttpUnit and EMMA integration.)

ReferencesSome other test and code coverage tools mentioned in the article are AgitarOne, dbUnit, Clover, Cobertura, HtmlUnit, JCoverage, Jester, Jtest, JUnitPerf, Selenium, XMLUnit (incomplete list).

(List of all my publications with abstracts.)

23 March 2009

Resources to Start Scala

The Scala Programming Language has been around some time and started getting popular in 2007. A good place to start are the tutorials included in the distribution, which are also available online: A Scala Tutorial (very short 15 pages), Scala By Example (already 145 pages) and The Scala Language Specification (full 180 pages).

Early multimedia resources that picked my interest were Martin Odersky's talk, The Scala Experience at JavaOne 2007 and the 62th episode from Software Engineering Radio. Further Martin Odersky gave another talk at JavaOne 2008 and JAX'08.

In January 2008 Ted Neward began his busy Java developer's guide to Scala, which started with stuff from the Scala Tutorial but went into greater detail later. Another blogger that is definitely worth mentioning is Daniel Spiewak, who wrote the nice Scala for Java Refugees as well as on some special topics like Integrating Scala into JRuby. Another piece worth recommending is Dean Wampler's blog titled the The Seductions of Scala. James Iry has to be mentioned for exploring more theoretical stuff in nice little chunks.

After spending some time with Scala, I went for the only book available, Programming in Scala, a comprehensive step-by-step guide with massive 754 pages. Unfortunately it did not ship for almost 9 (!) months and I do not like ebooks. (I know - I should have read the whole page when ordering; that it was not printed, not even finished back then.) However, now it's printed. Since January it's standing on my book shelf and torturing my conscience.

I have to pull myself together and finally start reading it!

19 January 2008

Daily Build Articles

In 2007 I started writing articles about Daily Builds for Java applications, which I called the 'Code Cop' Series. Unfortunately I just managed to finish two articles so far, shown below. I have a lot of material for further articles about adding automated testing and enforcing architecture to our daily build, just have to squeeze in the time to do it ;-)

The Daily GrindPart 1 - Daily Build with Anthill OS
This article describes my experiences when introducing a daily build in 2004 when I used the Anthill tool. The first steps were to create JavaDoc pages daily and to compile the Java sources. It turned out that the initial set-up of these build routines did not cost much and were supported by the team. Obviously this is only the start of better quality code. Read more in the full article Täglicher Build mit Anthill OS published in JavaSPEKTRUM, 5/2007.

ReferencesOther build tools are Anthill Pro, Continuum, Cruise Control, Maven, Luntbuild (incomplete list).

Part 2 - Daily Code Analysis with PMD
This article introduces static code analysis with PMD. The existing daily build was extended easily. A daily report of the code quality metrics awakened the management and was used as a base to check for a small set of errors. The most serious of them were fixed and part of the coding conventions have been checked automatically since then. Read more in the full article Tägliche Code-Analyse mit PMD published in JavaSPEKTRUM, 1/2008.

ReferencesTurn on the magicOther code analysis tools are Checkstyle, Enerjy CQ2, Findbugs, JavaNCSS, JLint, Jtest, Optimal Advisor, Sotograph (incomplete list).

(Download source code of Ant-PMD integration and BuildStatistics.java.)

FAQ
Q: Why did you favour Anthill OS instead of all other build tools listed? A: There was no evaluation or decision process. Anthill OS was just the first tool I got my hands on. Everything worked fine, so I did not look further. For a new project I would use Cruise Control because it is actively maintained and has a strong community. Paul Duvall has written a nice comparison of open source CI servers. If your build process is "heavy", you might want to have a look at commercial build servers. Some of them offer build farms. e.g. Anthill Pro, Parabuild or Team City.

Q: You divided all rules of PMD into four groups: error, warning, information and formatting. How can I get this categorisation? A: PMD comes with build in severity. Each rule definition contains an <priority> element. There are also some commercial tools that use PMD under the hood and have their own severity levels. Some of them even have references for each rule, why it's bad.

Q: You use a program called BuildRuleDoc to document used rules. Is it free or home-grown? A: I wrote it myself, but you can use it if you want to. The BuildRuleDoc.zip contains the code, the template to create the rule document, an Ant fragment and some test scripts. You will have to adapt the scripts in order to run them. Finally you need the XML rule set file of active PMD rules to generate the report.

(List of all my publications with abstracts.)

30 August 2006

Java Bookshelf

On this page you will find what I think are the most valuable resources for Java developers available. First a list of books every Java software developer should know (no excuses). It's amazing how old some of these are and still there are people around not knowing them. Next are my favourite newsletters (electronic journals), which I read regularly. Last I give you a (probably subjective) list of some of the best development tools available, which are all free.

Books
Books on the Java Language
Newsletters and Sites
Tools