Wednesday, September 5, 2018
#
This is the second of a two part blog post on code reviews. This second post is a technical checklist on some things to look for during a code review.
This is a 2 part blog post on code reviews. This first post is on why and when code reviews should be done and some of the soft skills needed.
Wednesday, September 20, 2017
#
So is it better to store enumerated types in a database as a string/varchar or integer? Well it depends, but in general as a string is your best bet. In this post I explore the pros and cons of each.
Lists verse Enums
Before we get to that let’s first be clear that I’m talking about enums here, not lists. Let me explain the difference.
For example let’s say to have a list of a available weight units: pounds, kilograms, grams, short tons, metric tons, long tons, stones, ounces, and etcetera. You might be able to design your database so that the list of possible weight units is in a table.
Your application should not have any advanced knowledge of the weight units that are defined in this table. The application reads this dynamic list at run time. Any information needed about these weight units must come from the database. This includes conversion factors, display names with multilingual support, flags to indicate in which situations it is appropriate to offer these units as a choice, mapping rules to external systems, and anything else the application may need. There should be no requirements that pounds must exist as a row or that kilograms must always be ID #1.
If you can soft code every behaviour of the weight unit required by applications in the database then the best design is to soft code this in a table. Installers may add or remove rows from this table as needed. This is a “List” of available values, which is defined in a database table.
If the available weight units is hard coded into the application as an enum or strongly typed enum then this is an “enum” not a “list”. The available values cannot change without changes to the programs. Therefore the available values should not be a database table. Adding support for new weight units requires code changes, so it is to the code that you must go to make this change. Having a database table falsely implies that the values are easy to change.
Lookup Times
The main argument for storing enum values as strings instead of integers is for human readability. People looking at the database can easily see what the values mean. Which of these two tables is easier to read?
| OrderID | Weight | WeightUnits |
| 1 | 14 | kg |
| 2 | 23 | lb |
| 3 | 25 | kg |
| 4 | 11 | lb |
| 5 | 18 | kg |
| OrderID | Weight | WeightUnitID |
| 1 | 14 | 1 |
| 2 | 23 | 2 |
| 3 | 25 | 1 |
| 4 | 11 | 2 |
| 5 | 18 | 1 |
Storing the values as a string makes it immediately obvious to anyone looking at the database what the units are. There is no need have external lookup tables either as database tables, database functions, or external documentation. It simply makes the system faster and easier to use.
The principle is similar to how applications should be developed so that users can get around with a minimum number of clicks. Adding just a few hundred milliseconds to the load time of a web page can cost thousands of dollars of lost sales. Even if the only people looking at the database are your own employees (i.e. a “captive audience”) the lookup time is still burdensome. Even if users have the IDs memorized, there is still a non-zero amount of time that it takes for the mental jumping jacks to make this conversion. There is a real cost to this lookup.
Systems should be designed for human readability, not computer readability.
Validation
String enum columns usually offer better validation and data quality. If someone enters a “jf” instead of “kg” for the units it is very obvious that a mistake has been made. Whereas fat fingers on the number pad entering pressing 2 instead of 1 is much harder to detect.
Having a 1 digit password is not a good idea. Making your password at least 2 or 3 characters gives much better reassurance that a monkey banging on the keyboard isn’t going to happen upon a valid, yet totally random and incorrect value.
Regardless on whether you choose to have enum values stored as integers or strings, it is a good idea to add a check constraint on the column to limit the values. In SQL Server, the T-SQL code to add a check constraint is as follows:
Alter Table Orders
Add Constraint CHK_Orders_WeightUnits Check(WeightUnits In ('kg', 'lb'))
There are also pros and cons to adding the check constraint to the database. The downside is that if you want to enhance your program to support a new enum value you must modify both the program and the database constraint.
You need to time the upgrades to the application and database. In general, update the programs to support the new values if the are read from the database, then update the database constraint to allow the new values, then update the applications (or feature toggle) to allow the new values to be written to the database.
Performance
I do acknowledge that using integers over strings gives better performance. Databases are smaller, less data is transferred over the network, and in theory faster. Exactly how much faster? I’m not sure, I haven’t run the benchmarks. Even with storing enums and strings I aim to have all my queries run in a few milliseconds. In most cases, I would categorize using integers over strings to be a micro-optimization.
However, when creating the enum string values do try to use as little space as possible. You don’t want to make your enum values dozens of characters long. Always try to keep these values less than 10 characters. Try to use abbreviations to reduce the values further down to 2 or 3 characters. Always use varchar instead of nvarchar.
For normal table columns I don’t think there is a significant performance hit in using strings. However, one place I might consider using an integer is if the column was used as part of an index. Database indexes are meant to make databases go fast, therefore every effort should be made to optimize these.
However, it is rare that enum (or Boolean) columns are used in indexes. In general those types of columns don’t make for good indexes. The goal of a database index is to take a table with a million rows and narrow that row count down to just the few rows you are searching for. Dates and free text (e.g. a person’s last name) are good candidates for indexes because they drill down from millions of rows to just a few - quickly. With an enum with 5 possible values, including the enum in the index is only going to reduce the millions of rows by a fifth (still hundreds of thousands).
If there is an enum value that does significantly reduce the row count, then consider using a filtered index. Put the enum value in the filter and not directly in the index. In that case the string enum value isn’t stored in the index; it is just used to decide which rows to put in the index.
Create Index IX_Orders_StatusUnderReview
On dbo.Orders(OrderID)
Where (Status = 'Review')
With this query the “Select * From Orders Where (Status = ‘Review’)” does use this index even though the Status column is not an indexed column.
Thursday, February 23, 2017
#
So I gave a talk at the Edmonton .NET User Group on an introduction to automated testing. And of course in this introduction I gave a list of reasons why unit testing is a good idea. Although there are already many blog posts other there on this subject, I thought I’d record my reasons here.
Although all of these reasons may appeal to your hipster, idealistic sense of craftsmanship and professionalism, each of these reasons also has a sound business foundation. Automated testing saves/makes you money.
1. Product Quality
Of course the most obvious reason for automated testing is to improve product quality. Although automated testing will not guarantee that you’ll never have another bug, it will lower that bug rate. And having working software is the #1 design goal. Telling customers about the automated testing that is in place can be a strong marketing point.
Nobody will buy software that doesn’t work.
2. Because you are doing it anyway
If you aren’t doing automated testing, then you are doing manual testing. Manual testing (and all testing) is a thankless job that doesn’t get the credit it deserves. A lot of people assume it just happens for free.
Manual testing does take a lot of time and effort. To run a test takes a lot of time just to set up. For example, confirming how the system behaves with a customer with bad credit. You have to create or find a customer record and change its credit status. Once the setup is done comes the actual testing. You have to run the program, login as the customer, and attempt to make a purchase. This all takes time, several minutes.
Setting up an automated test also takes quite a bit of time. This is especially true for the first test. However the good news is you only have to do this once. Once the test is written you can run it as often as you want, several times an hour typically. Also running the test is much faster; no having to start the program, login, and all the other typing and button clicks. And the code you write to setup your first test will be reused for all the other tests.
Although the first test is the most difficult to write, before the first production build of your program the time spent building an automated test suite will be less that then time it takes to manually test the same program once. And once the automation is in place, it can be run whenever virtually instantaneously.
Automated testing takes less time than manual testing. This is certainly true in the full lifecycle of the program, but is even true before the first production release.
3. Enables Refactoring
Refactoring is the practice of going back over existing code and improving the readability of it. It removes technical debt.
A common practice on my team is doing code reviews, especially with new hires. The result of code review is often a list things that should have been done differently. On receiving these suggestions the programmer asks, should I implement those changes. The answer is, it depends.
If the testing has been done manually then the answer is often “no”. The code review is at the end of the project and we are up against a deadline. Although the changes may be fairly simple, the effort to manually retest the program does not out weight benefits of the cleaner code. It is usually just too risky to change the program.
This is true not only for the first release, but future maintenance cycles. When other developers go into the same code base and need to make a minor enhancement. The risk that changing the program will introduce a bug is too high.
Furthermore, having a single program contain two design styles is a sin unto itself. New features will likely be implemented following the example of the design already in place. The bad design will perpetuate. This is similar to the Broken Windows Theory. The code begins to rot. Eventually it reaches the point where the only course of action is to throw the existing code away and begin a full rewrite.
However, if the program supports an automated test suite the answer to if the code should be improved is usually “yes”. The very first time I used unit testing we did the normal code review process and the experience was completely different than previously. Even though the code review was done only hours before the deadline, most of the suggestions were implement before the release. This was because changes could be made without the penalty of a long testing cycle and risk of introducing a bug.
The quality of code for a program without unit tests will only degrade over time. A program with unit tests will improve in quality over time. This improves the longevity of the program and avoids “the big rewrite”.
4. Better Design
This point is a little controversial, but in general unit testing will improve the design of a program. Code that is testable will be designed to be smaller. It will yield code that better follows the Single Responsibility Principal. It will be less coupled to other parts of the system and have better reuse potential.
Here is an excellent video by Michael Feathers on why and how unit testing encourages good design: https://vimeo.com/15007792. It states it much better than I can.
The counter argument is that sometimes this can cause systems to become over designed. This point was made by David Heinemeier Hansson in a blog post. This caused much debate on the internet, some of which was captured here between David and Kent Beck in this 5 part video series.
At the end of the day, the programmer is in control of the design of the system. Doing Test Driven Development is a tool that helps with design, but it does not take away decisions from the programmer. Automated testing forces programmers to think more diligently about design and makes them better designers.
5. Quicker Feedback Loop
Staying in “the zone” is critical to being productive while developing software. There are dozens of pieces of information and background context that need to be kept in mind while programming. When you switch from one task to another there is a productivity drop of 20%.
With manual testing often the developer implements a feature, compiles it a week later, it sits in the tester’s inbox for a week, tested, then the test results get analyzed by the developer a week later. It is common there is a 2 or 3 week gap between implementation and the time you need to fix it. When the test results are received it can take hours to reproduce the issue and find the offending line of code.
With unit testing the workflow is write a line or two of code, then run the tests. When a test fails it is simple to remember the line of code that caused the bug. That’s because you just wrote that line of code a minute ago.
Unit testing takes the debugging process from an hour or more to a minute or less. Unit testing adds productivity due to the short feedback loop.
6. Team Ownership
The great thing about automated testing is that once the test is created it is saved and shared with the entire team. Every other team member pull in that test when they do their testing.
If a developer implements another user case scenario that is in conflict with existing features the automated tests will reveal this immediately. All developers on the team must support all the features of the system, not just the small part of the system they are working on.
Automated testing ensures that older tests are always run (i.e. regression testing). This prevents breaking changes from affecting existing customers. It improves communication on the team because all these cases are known.
7. System Documentation
With all development, but especially with APIs, providing working examples of how the code should be called and what the expected results are is important. Automated tests can provide these examples.
Also the automated tests will never become stale like a English document may become. The tests are guaranteed to stay current with the production code because they are run with every build.
8. Fun
Manual testing is a boring, tedious job. Running the same test scripts over and over again is mind numbing. What is boring and tedious should be automated. This is what computers are good at.
Most developers find the challenge of automating testing to be enjoyable. Asking employees to do a job that is boring will likely cause them to quit.
And also, it is no fun having customers yell at you because the software doesn’t work. Bugs lead to blame. Blame leads to fear. Fear leads to anger. Anger leads to hate. Hate leads to suffering. ~Yoda
Automated testing decreases employee turnover.
9. Required Skill
When getting started with automated testing (or any new technology or process) there will be resistance. You will hear the fallacies that it will take too long, its too hard, or it will hurt the design. Whatever the reasons you have for not doing automated testing, don’t let that you don’t know how be one of them.
The harshest critic of automated testing will still conceded that in some circumstances it is still proper thing to do. Therefore when the time comes that testing should be automated ensure that you and your team are proficient in doing so. It is a skill that can only be learned from practice.
Monday, October 17, 2016
#
So I thought I’d write down some tips for using a chat application effectively on a development team. Chat applications can revolutionize a how a development team works and even its culture. I’ll go through some of the goals of implementing a chat application, some best practices, and pitfalls.
Goals
Obviously the goal of a chat application is to improve communication amongst the team. It is meant to remove “silos”. Silos refers to an organizational structure where members of the same team do not communicate directly with each other. The image of a silo is a very tall structure with a very small opening (for communication) at the top or bottom. Developers aren’t given good opportunities to communicate with each other except through their manager or what they deliver.
Silos are bad for development environments because often different members of the team will duplicate work. They aren’t aware that others have already solved the same problem. Or worse yet, the work of different team members can be divergent, pulling the products and development strategies is different, conflicting directions.
The answer to this in the Agile and Scrum movements is “The Daily Stand-up”. This is a brief meeting at that start of the day where everyone gets together to let the team know what they are working on. The typical structure of the meeting is to answer “3 questions”: What did I do yesterday, what will I do today, and what impediments are getting in my way. Just by answering these questions publically to the team can be very effective in breaking down silos.
One of the problems with the Daily Stand-Up is that it does take some coordination. Everyone has to get together at the same time and in the same physical space. The meeting can’t start until everyone arrives. This can be problematic if different people are on different schedules. Remote developers and different time zones, client phone calls, and traffic delays can all infer with this. A chat application can be a very low friction way to allow this meeting to happen with the flexibility for people’s schedules and locations.
It also allows for updates to be posted multiple times a day. If someone runs into an impediment they don’t have to wait until the next morning to seek the team’s help.
The chat application is well sized as a communication medium for this problem. Phone calls or face-to-face meetings can be highly disruptive. Developers are forced to drop what they are doing immediately. It can take from 10 to 25 minutes or more to recover from the interruption. Email on the other hand can become too long winded. It lacks the brevity that chat encourages. When email is used as chat the large number of chat emails can make drowned out the important emails. Chat seems to be a happy median between these extremes. It is brief and it can be easily deferred and non-interruptive when desired.
Chat can also improve the interpersonal relationships on a team. Instead of the only team communication being at a biweekly manager-supervised sprint planning meeting, communication occurs daily. Most developers know how disruptive face-to-face communication can be, therefore they avoid it. Chat is less formal, non-intrusive vehicle for developers to interact.
Getting Started – The Daily Stand-up
So to get started using a chat application on a developer team, I’d suggest starting with the Daily Stand-up 3 Questions: What I did yesterday, today, and impediments. That is the standard format for the physical stand-up meeting.
I don’t think that chatting about yesterday is needed, because all the chat history is there. But letting people know what you are about to work on and anything getting in your way is critical.
No Surprises
Chat can be used to post whenever a release is made of software. Most chat applications allow for these postings to be done automatically from the build process. This makes sure that everyone is aware of changes that have been made to the product.
Although learning what has been done is nice, it is best if the chat contains information about what is about to be done before it is done. Sometimes release announcements can be greeted with “you released what?!!”. Work items should be bookended in the chat application so that people are clear when work starts and ends. This way there are no surprises.
The main advantage of the chat application is informing the team what is being worked on at the earliest possible moment. This allows for feedback to be received as soon as possible and nip any problem in the bud. There are a number of advantages to announcing before work is done as opposed to after.
It corrects any communication errors around priorities. If you are working on the wrong thing, this will quickly be caught. This can also prevent yak shaving.
Chatting allows to catch duplicate work before it is done. I someone posts that they are working on a class to process command line arguments, other on the team can intervene and point to the location where that code is already written.
This process also works good for design. The chat application can serve as a good brainstorming tool. It can allow for feedback on designs in their infancy. Many think that designs are very mutable as long as they haven’t reached the implementation stage (coding) yet. However spending days designing and documenting a new feature can cement the mindset on the proposed design. The design review can become a violent defense of one person’s design rather than an open discussion. If feedback was allowed before the initial documentation is done the design much more fluid and open to change.
Chat works well after the design meeting as well. At that design meeting there is usually an agreement of how the code should be implemented. However when coding occurs you usually run into one or two minor details that weren’t thought though well enough in the design meeting. The design needs to be changed ever so slightly. It is really a small change and doesn’t warrant reconvening the entire design review committee. However over the course of a week or two there may be several of these very small design changes that begin to add up. The coders can post on the chat the small corrections they are making to the design so that no one is surprised.
When coding you are never writing the same code, solving the same problems day after day. Everyday you take the product is a slightly new direction or learn a new technique. You’re constantly faced with a couple different options. Each decision made should be broadcast so that the team has a common vision of the path being taken.
Conversation Starters
Not all chat posts are made because you are seeking immediate feedback. Sometimes they are a reminder for a future conversation. When I meet with friends or family and they ask, “what I’ve been up to?”, I often struggle for an answer. It is usually the standard “same old, same old, nothing much, work and home”. However, if I share Facebook with them the posts often spawn a conversation. Didn’t you go camping or to a concert a few weeks ago? How was that? Using chat on a development team can be similar. Weren’t you working on some encryption algorithm? How did that turn out?
Today I Learned
“Today I Learned (TIL)” posts are also great for chat. If you learned something new today then it is likely that there is at least one other member of the team that doesn’t know that either.
Chat is good for just educate other team members what your dark corner of the system. You may be working on components that some members on the team might not even know exist. Use the chat to teach others about the system.
Estimation Aide
To get better at something you need to practice. If you are currently estimating work at biweekly sprint planning meetings then that is 26 opportunities to practice estimating per year. If you are posting daily or twice a day what you estimate you’ll be able to complete in the next few hours then that is hundreds of estimates per year.
Special Roles
Chat can be great for monitoring support calls as well. It can give any developer insight into the types of problems that customer support is dealing with. Reoccurring issues can be noted and preventative action taken.
Chat can also be helpful for on ramping new developers. The new hire will get the sense that the entire team is behind them, instead of just a single mentor. Chat exposes the training given and ensures there is no misinformation or inconsistencies.
Anti Patterns
If using Chat as a daily stand-up meeting there is the potential for the same problem. That is “zombie scrum”. The answers to the 3 questions become a dull drone that is repeated day after day, “I’m working on the same thing today as I did yesterday, no impediments”. If you are listing your current work item as the same thing day after day, you are not being descriptive enough in your post.
“No impediments” listed day after day is also an anti-pattern. If you haven’t posted any impediments in the last few weeks, then you might want to examine your chat history for an “elephant in the room”. I’m going to spend all day today resetting my test environment, and I have no impediments. Well maybe the fact that you’re spending all day to create a test environment and that step is not automated is the impediment.
Be clear and informative on what you are working on. Don’t just state the program or component you are working on. What enhancement is being made? What value will that bring to the customer?
Don’t simply quote bug numbers or source control check-in IDs. Don’t say read my design document or check the minutes of the meeting if you want details. Give some details on the work that is being done. Describe what decisions are being made about how something is being implemented. Give some quick highlights of any external references.
Be respectful. Don’t use the chat for public hazing. If you think your comments might be considered overly critical, discuss them privately first. Let the original poster correct or clarify their statements instead of being corrected by others.
Sometimes chat or other meetings can get in the way of getting work done. It is OK to turn off the chat application, email, and take the phone off the hook to get a task done. Most chat applications do have a “do not disturb” feature.
However, do remember to turn it back on when you are ready to reengage with the team. Review the posts that happened while you were dark. Obviously, a way the chat system can fail is if nobody uses it. Another way it can fail is if people just “lurk”. They will read what people are posting, but not offer up any comments of their own. They may be reluctant to post out of fear of saying something stupid. It might be an impostor syndrome, where they think that anything they have to post everyone else probably already knows. In order for Chat to be successful, you must be active.
Summary
- Post what you are working on
- Post decisions being made
- Post anything learned
- Don’t repeat yourself
Happy Chatting.
Monday, October 26, 2015
#
So I was listening to a recent Western Developer’s podcast on Ethics. At one point in the conversation [39:30] there was a call for establishing a industry standard professional body for software developers. This would be much like the Bar Association for lawyers or the Medical Board for doctors. It would be responsible for establishing a set of guidelines or rules for software development professionals to follow. It would grant membership to those properly trained and punish those that act unprofessionally. This is also a common theme from Uncle Bob [23:00].
Neither of these podcasts stated what that code of ethics should be or how we would go about forming this body. It would certainly be difficult to do. In my opinion this will never happen. I hold this opinion not because I think it is too difficult, but because I think we shouldn’t. I do believe we need to act professionally and we need to apply ethics to our jobs. However, I don’t think we could successfully build a governance body and have this widely accepted by our clients.
We are a service industry. We generally don’t serve individuals, like doctors serve patients. We serve industries. We write software for the health care, legal, engineering, or accounting industries. We usually specialize in serving a single industry. In my opinion software companies that specialize in a single industry are more successful at meeting their clients needs.
When specializing in servicing an industry you must become part of that industry. You must learn to talk like you clients. You must adopt their culture and behaviours.
For example let’s look at serving the health care industry. Doctors are bound by their medical board to save lives. That is their code of ethics. Although hospital administrators are not doctors and not ruled by the same medical board, due to their service to the industry they do follow many of the same core ethics. Hospital administrators work within fixed budgets to save as many lives as possible. Their actions save lives, the same as doctors save lives.
A hospital administrator could print flyers reminding people to wash their hands and this can save lives. The costs of this printing project could save more lives per dollar than keeping a surgical staff on call. So when an administrator needs to make a decision to print flyers or fund a surgical team they can apply their ethics to help them make this decision.
Now for a software example, let’s say a hospital administrator wants you to build a big design up front, 3 year waterfall project. You know because of your training and/or experience as a software development professional that this approach is likely to fail. It is a waste of money. A professional software developer would argue with the administrator that it is better to break the project down into smaller Agile projects.
When arguing with the hospital administrator as to the whether or not the waterfall project should be done, this argument must be framed in the context of the health care industry. The software developer should refuse to do the waterfall project. Not because their software professional organization states that waterfall is wrong. The reason should be because the waterfall project will not save as many lives compared to spending that same hospital budget on other potential projects.
We cannot superimpose ethics of the software development profession (or any other industry) on our client’s industry. If we cannot restate our ethics in the terminology of our client’s industry we will not be successful.
In the accounting industry there is a very strict audit trail. Every penny received from the sale of an item can be traced through to how that is redistributed to employee salaries, building maintenance costs, raw materials, vendors, taxes, and shareholders. An accountant would consider it unprofessional if they were not able to produce this audit trail.
However, other industries do not have as strict an audit trail. The agriculture and food industry generally cannot produce an audit trail to trace a loaf of bread back to the farmer’s field where the wheat was grown on. An accountant may look at that and claim it is unprofessional. The farmer and baker would reject their arguments. The accountant would not be able to argue the point purely on the professionalism of an audit trail. However, they might be more successful if they could reframe their argument as a food safety issue. Food safety is part of the ethics the farmer and baker are bound to. The agriculture industry does batch together food from multiple farms and verify the quality and safety of the batch, but they might never be able to trace grains of wheat as accurately as accountants track pennies.
Furthermore, to the agricultural professional one should not waste food or other resources. Tracking individual grains of wheat would certainly be seen by farmers and bakers as wasteful and therefore unprofessional. The same qualities that may make someone a professional in one industry could seem unprofessional if unconditionally applied in other industries.
Although we can and should train software developers to be professional and ethical, we cannot directly superimpose those standards on the industries we serve. Professional software developers must hold two degrees: one in software development and another in the domain and ethics of the industry they serve. If we cannot adapt our ethics to the terms of our clients (on a client by client basis) we risk being branded as unprofessional. This could, or perhaps has, lead to the software profession as a whole being labeled as immature.
Friday, February 20, 2015
#
Review of the Simplicity principles and anti-patterns: KISS, Clever Code, Golden Hammer, Yak Shaving, YAGNI, Premature Optimization, Worse Is Better, Big Design Up Front, and Bike Shedding.
Tuesday, February 17, 2015
#
I believe that Beck’s Four Rules can be simplified to 2 axioms:
1. Write working software
2. Write simple software
Monday, September 1, 2014
#
The behaviour we want out of this pattern is when a connection error occurs we should queue up a notification to be delivered at some point in the near future. When it comes time to actually deliver the notification we check if the connection has been restored. If the connection has been restored we cancel the notification.
A feature that every serious program should have is the ability to notify when an unexpected error occurs. Every unexpected error should be reported and investigated. This is necessary in order to build quality software. In this post I’ll cover an Rx notification pattern that will report unexpected errors.